2025-02-13 Thu 00:50:26:647.978 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: http-server-port 8091 2025-02-13 Thu 00:50:26:648.083 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Max-tasks 16 2025-02-13 Thu 00:50:26:648.095 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Max-slice 100 2025-02-13 Thu 00:50:26:648.104 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Collectors 10.0.0.46:8086 2025-02-13 Thu 00:50:26:648.111 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Collectors 10.0.0.35:8086 2025-02-13 Thu 00:50:26:648.118 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Collectors 10.0.0.12:8086 2025-02-13 Thu 00:50:26:648.144 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: ROLE : Generator 2025-02-13 Thu 00:50:26:648.153 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-13 Thu 00:50:26:648.160 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2728-2. 2025-02-13 Thu 00:50:26:648.168 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-13 Thu 00:50:26:648.175 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-13 Thu 00:50:26:648.183 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-13 Thu 00:50:26:648.192 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-13 Thu 00:50:26:648.817 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: HTTP Introspect Init 2025-02-13 Thu 00:50:26:648.925 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-13 Thu 00:50:26:648.982 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Sandesh Http Server Port: 8091 2025-02-13 Thu 00:50:26:649.021 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-13 Thu 00:50:26:649.030 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-13 Thu 00:50:26:649.114 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Idle 2025-02-13 Thu 00:50:26:649.273 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: No Client: 1739407826649196 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:50:26:649.402 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-13 Thu 00:50:26:649.416 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-13 Thu 00:50:26:649.428 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Cassandra Servers: 10.0.0.46:9042 10.0.0.35:9042 10.0.0.12:9042 2025-02-13 Thu 00:50:26:649.522 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Processing scm::EvStart in state Idle 2025-02-13 Thu 00:50:26:649.593 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Disconnect 2025-02-13 Thu 00:50:26:649.666 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-13 Thu 00:50:26:649.875 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Connect : Start Connect timer 10.0.0.46:8086 2025-02-13 Thu 00:50:26:649.947 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:649.968 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: SANDESH: Send FAILED: 1739407826649584 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-13 Thu 00:50:26:649.977 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:50:26:649.986 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:650.075 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: SANDESH: Send FAILED: 1739407826649655 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:50:26:650.092 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:50:26:650.102 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:650.199 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: SANDESH: Send FAILED: 1739407826649714 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:50:26:650.215 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:50:26:650.225 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:650.240 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: SANDESH: Send FAILED: 1739407826649735 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-13 Thu 00:50:26:650.247 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:50:26:650.255 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:650.318 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: SANDESH: Send FAILED: 1739407826649936 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:50:26:650.331 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:50:26:666.771 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-13 Thu 00:50:26:666.816 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:26:666.887 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:26:666.926 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968693438208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:666.957 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968693438208, Pid 1]: SANDESH: Send FAILED: 1739407826666856 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-13 Thu 00:50:26:666.966 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968693438208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:50:31:670.788 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-13 Thu 00:50:31:670.836 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:31:670.875 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:36:672.784 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-13 Thu 00:50:36:672.835 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:36:672.858 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:41:674.899 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-13 Thu 00:50:41:674.951 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:41:674.982 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:46:895.048 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:50:46:895.107 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:46:895.131 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:51:931.092 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:50:51:931.146 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:51:931.169 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:56:962.080 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:50:56:962.138 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:56:962.162 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:01:986.325 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:51:01:986.390 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:01:986.449 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:07:025.217 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:51:07:025.299 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:07:025.330 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:12:055.593 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:51:12:055.666 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:12:055.691 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:17:090.042 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:51:17:090.114 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:17:090.137 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:22:172.357 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:51:22:172.419 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:22:172.451 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:27:196.884 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:27:196.963 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 12 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:32:221.704 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-13 Thu 00:51:37:235.291 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:51:37:235.327 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 00:51:37:235.634 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 00:51:37:235.667 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x12511a0 2025-02-13 Thu 00:51:37:235.674 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:235.821 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:235.414 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SANDESH: Send FAILED: 1739407897235188 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-13 Thu 00:51:37:235.841 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x1251ae0 2025-02-13 Thu 00:51:37:235.867 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:235.893 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:235.913 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 00:51:37:235.941 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 00:51:37:235.944 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:235.991 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x124f5f0 2025-02-13 Thu 00:51:37:236.002 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.015 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.026 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x1252420 2025-02-13 Thu 00:51:37:236.032 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.041 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.051 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x1252d60 2025-02-13 Thu 00:51:37:236.083 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.091 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.101 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x12491d0 2025-02-13 Thu 00:51:37:236.111 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.102 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 00:51:37:236.122 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.150 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x124fca0 2025-02-13 Thu 00:51:37:236.155 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.164 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.170 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Idle 2025-02-13 Thu 00:51:37:236.174 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x12536a0 2025-02-13 Thu 00:51:37:236.198 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.206 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.217 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x1250340 2025-02-13 Thu 00:51:37:236.223 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.233 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.243 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x1250870 2025-02-13 Thu 00:51:37:236.249 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.249 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:236.260 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.286 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x1253fe0 2025-02-13 Thu 00:51:37:236.291 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.300 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.314 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x1255260 2025-02-13 Thu 00:51:37:236.319 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.339 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.349 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x1254920 2025-02-13 Thu 00:51:37:236.355 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.365 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.377 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SANDESH: Send FAILED: 1739407897236041 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 42 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:51:37:236.401 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:51:37:236.380 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x1255ba0 2025-02-13 Thu 00:51:37:236.415 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:236.430 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.455 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SANDESH: Send FAILED: 1739407897236080 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-13 Thu 00:51:37:236.470 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.472 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:236.492 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: RAC_Connect status: 0 0x12564e0 2025-02-13 Thu 00:51:37:236.495 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 00:51:37:236.506 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:236.540 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:236.551 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:236.662 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SANDESH: Send FAILED: 1739407897236240 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 47 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 49, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:51:37:236.678 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:51:37:236.764 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:236.891 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SANDESH: Send FAILED: 1739407897236702 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739407897236169 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739407897236549 ] max_sm_queue_count = 5 ] 2025-02-13 Thu 00:51:37:236.902 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:51:37:236.912 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:237.012 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SANDESH: Send FAILED: 1739407897236751 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 54 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 56, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:51:37:237.035 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:51:37:237.083 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:237.120 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:237.165 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:237.238 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:237.262 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:237.285 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:237.307 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:237.377 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:237.399 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:237.413 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:237.540 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:237.560 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:237.590 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:237.605 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: SANDESH: Send FAILED: 1739407897237499 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 00:51:37:237.615 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:237.633 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:237.619 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:237.653 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: SANDESH: Send FAILED: 1739407897237579 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.46:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 00:51:37:237.667 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:237.672 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:237.721 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:237.755 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:237.811 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:237.848 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:237.864 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:237.883 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:237.897 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:237.935 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:237.949 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:237.967 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:238.002 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:238.022 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:238.037 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:238.077 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:238.099 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:238.118 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:238.132 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:238.187 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:238.246 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:238.275 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:238.286 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:238.312 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: SANDESH: Send FAILED: 1739407897238222 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 00:51:37:238.321 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:238.325 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:238.352 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:238.387 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:238.475 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968693438208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:238.478 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:238.512 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968693438208, Pid 1]: SANDESH: Send FAILED: 1739407897238464 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 00:51:37:238.516 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:238.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968693438208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:238.666 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:238.687 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:238.711 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:238.728 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:238.777 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:238.797 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:38:236.350 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 00:51:38:236.686 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Connect : Start Connect timer 10.0.0.35:8086 2025-02-13 Thu 00:51:38:236.824 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:51:38:236.862 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: SANDESH: Send FAILED: 1739407898236456 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 00:51:38:236.872 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:38:236.881 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:51:38:237.014 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: SANDESH: Send FAILED: 1739407898236807 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 101, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:51:38:237.029 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:51:38:238.956 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:56514::10.0.0.35:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 00:51:38:238.983 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: 10.0.0.35:56514::10.0.0.35:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 00:51:38:238.991 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: 10.0.0.35:56514::10.0.0.35:8086(-1):EvTcpConnected 2025-02-13 Thu 00:51:38:239.055 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 00:51:38:239.078 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 00:51:38:239.110 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: ClientInit 2025-02-13 Thu 00:51:38:239.129 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-2.:contrail-query-engine:0:Database count 1 2025-02-13 Thu 00:51:38:239.206 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 00:51:38:239.215 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 00:51:38:239.238 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: SANDESH: Send FAILED: 1739407898239104 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 00:51:38:239.246 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 00:51:38:239.255 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 00:51:38:239.262 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 00:51:38:239.272 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 00:51:38:239.286 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 00:51:38:239.396 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: SANDESH: Send FAILED: 1739407898239195 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 102, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:51:38:239.410 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 00:51:38:240.666 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: OnMessage control in state: ClientInit 2025-02-13 Thu 00:51:38:240.700 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-13 Thu 00:51:38:240.709 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-13 Thu 00:51:38:240.732 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: Received Ctrl Message with size 0 2025-02-13 Thu 00:51:38:240.940 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739407897236169 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739407897236549 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 102, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2025-02-13 Thu 00:51:38:240.972 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 00:51:38:240.996 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-02-13 Thu 00:51:38:241.010 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 00:51:38:241.021 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-13 Thu 00:51:38:241.028 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 00:51:38:241.055 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: Established 2025-02-13 Thu 00:51:48:501.842 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: OnMessage control in state: Established 2025-02-13 Thu 00:51:48:501.896 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-13 Thu 00:51:48:501.907 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Established : EvSandeshMessageRecv 2025-02-13 Thu 00:51:48:501.949 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: Received Ctrl Message with size 3 2025-02-13 Thu 00:51:48:501.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-02-13 Thu 00:51:48:501.976 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-02-13 Thu 00:51:48:501.988 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-02-13 Thu 00:51:48:501.996 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 00:52:07:236.990 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-13 Thu 00:53:07:237.736 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968403982080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-13 Thu 00:54:07:238.796 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-13 Thu 00:54:34:117.776 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:8091::10.0.0.35:53218 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-13 Thu 00:55:06:425.805 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968837109888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:8091::10.0.0.35:56320 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-13 Thu 00:55:07:239.344 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-13 Thu 00:56:07:239.887 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968391386880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-13 Thu 00:57:07:240.474 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968399783680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-13 Thu 00:58:07:241.046 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968399783680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-13 Thu 00:59:07:241.631 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-13 Thu 01:00:07:242.162 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-13 Thu 01:01:07:243.276 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-13 Thu 01:02:07:243.813 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968403982080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-13 Thu 01:03:07:244.422 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-13 Thu 01:04:07:246.712 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968650794752, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-13 Thu 01:05:07:247.543 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968403982080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-13 Thu 01:06:07:248.655 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-13 Thu 01:07:07:249.239 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968650794752, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-13 Thu 01:08:07:250.304 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968399783680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-13 Thu 01:09:07:250.895 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968403982080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-13 Thu 01:10:07:253.077 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968638203648, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-13 Thu 01:11:07:253.663 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968399783680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-13 Thu 01:12:07:254.202 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968403982080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-13 Thu 01:13:07:254.746 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968391386880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-13 Thu 01:14:07:255.259 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968403982080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-13 Thu 01:15:07:256.482 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-13 Thu 01:16:07:257.116 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-13 Thu 01:17:07:257.693 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968391386880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-13 Thu 01:18:07:258.304 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968659187456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-13 Thu 01:19:07:259.320 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-02-13 Thu 01:20:07:260.343 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-02-13 Thu 01:21:07:260.920 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968638203648, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-02-13 Thu 01:22:07:261.573 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-02-13 Thu 01:23:07:262.169 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968638203648, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-02-13 Thu 01:24:07:262.768 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-02-13 Thu 01:25:07:263.304 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968672446208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-02-13 Thu 01:26:07:263.930 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-02-13 Thu 01:27:07:264.500 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-02-13 Thu 01:28:07:265.783 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968659187456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-02-13 Thu 01:29:07:266.363 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-02-13 Thu 01:30:07:267.033 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968638203648, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-02-13 Thu 01:31:07:267.659 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-02-13 Thu 01:32:07:270.333 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-02-13 Thu 01:33:07:271.108 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968399783680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-02-13 Thu 01:34:07:271.707 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-02-13 Thu 01:35:07:272.210 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968638203648, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-02-13 Thu 01:36:07:272.784 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-02-13 Thu 01:37:07:273.849 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-02-13 Thu 01:38:07:274.335 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968391386880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-02-13 Thu 01:39:07:275.029 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-02-13 Thu 01:40:07:275.590 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-02-13 Thu 01:41:07:276.129 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-02-13 Thu 01:42:07:276.720 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-02-13 Thu 01:43:07:277.828 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968680843008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-02-13 Thu 01:44:07:278.803 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968391386880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-02-13 Thu 01:45:07:279.667 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968399783680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-02-13 Thu 01:46:07:280.268 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-02-13 Thu 01:47:07:281.368 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968689239808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-02-13 Thu 01:48:07:281.956 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968659187456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-02-13 Thu 01:49:07:282.582 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-02-13 Thu 01:50:07:283.186 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968399783680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-02-13 Thu 01:51:07:283.885 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968391386880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 60 without seqno , total = 0 2025-02-13 Thu 01:52:07:284.452 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 61 without seqno , total = 0 2025-02-13 Thu 01:53:07:285.071 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968399783680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 62 without seqno , total = 0 2025-02-13 Thu 01:54:07:285.581 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968399783680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 63 without seqno , total = 0 2025-02-13 Thu 01:55:07:286.728 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968659187456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 64 without seqno , total = 0 2025-02-13 Thu 01:56:07:287.315 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968412378880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 65 without seqno , total = 0 2025-02-13 Thu 01:57:07:287.820 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968659187456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 66 without seqno , total = 0 2025-02-13 Thu 01:58:07:288.348 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968408180480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 67 without seqno , total = 0 2025-02-13 Thu 01:59:07:289.022 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 68 without seqno , total = 0 2025-02-13 Thu 02:00:07:289.572 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968680843008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 69 without seqno , total = 0 2025-02-13 Thu 02:01:07:290.089 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968685041408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 70 without seqno , total = 0 2025-02-13 Thu 02:02:07:290.746 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 71 without seqno , total = 0 2025-02-13 Thu 02:03:07:291.273 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 72 without seqno , total = 0 2025-02-13 Thu 02:04:07:291.809 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968659187456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 73 without seqno , total = 0 2025-02-13 Thu 02:05:07:292.398 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968642402048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 74 without seqno , total = 0 2025-02-13 Thu 02:06:07:293.062 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968676644608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 75 without seqno , total = 0 2025-02-13 Thu 02:07:07:293.886 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968659187456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 76 without seqno , total = 0 2025-02-13 Thu 02:07:39:918.885 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968650794752, Pid 1]: Processing scm::EvStop in state Established 2025-02-13 Thu 02:07:39:918.968 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968650794752, Pid 1]: Idle 2025-02-13 Thu 02:07:39:919.078 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968650794752, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:07:39:919.090 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968650794752, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:07:39:919.235 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968650794752, Pid 1]: SANDESH: Send FAILED: 1739412459919066 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739407826649125 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3688250 calls = 385 average_bytes = 9579 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 76 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 306 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 104, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 386, ] ] ] 2025-02-13 Thu 02:07:39:919.258 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 139968650794752, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:07:57:498.985 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: http-server-port 8091 2025-02-13 Thu 02:07:57:499.095 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Max-tasks 16 2025-02-13 Thu 02:07:57:499.104 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Max-slice 100 2025-02-13 Thu 02:07:57:499.110 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Collectors 10.0.0.46:8086 2025-02-13 Thu 02:07:57:499.115 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Collectors 10.0.0.35:8086 2025-02-13 Thu 02:07:57:499.120 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Collectors 10.0.0.12:8086 2025-02-13 Thu 02:07:57:499.141 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: ROLE : Generator 2025-02-13 Thu 02:07:57:499.148 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-13 Thu 02:07:57:499.153 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2728-2. 2025-02-13 Thu 02:07:57:499.159 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-13 Thu 02:07:57:499.164 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-13 Thu 02:07:57:499.169 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-13 Thu 02:07:57:499.175 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-13 Thu 02:07:57:499.781 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: HTTP Introspect Init 2025-02-13 Thu 02:07:57:499.902 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-13 Thu 02:07:57:499.962 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Sandesh Http Server Port: 8091 2025-02-13 Thu 02:07:57:499.995 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-13 Thu 02:07:57:500.005 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-13 Thu 02:07:57:500.083 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Idle 2025-02-13 Thu 02:07:57:500.244 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: No Client: 1739412477500167 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:07:57:500.381 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-13 Thu 02:07:57:500.395 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-13 Thu 02:07:57:500.407 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Cassandra Servers: 10.0.0.46:9042 10.0.0.35:9042 10.0.0.12:9042 2025-02-13 Thu 02:07:57:505.373 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Processing scm::EvStart in state Idle 2025-02-13 Thu 02:07:57:505.447 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Disconnect 2025-02-13 Thu 02:07:57:505.525 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-13 Thu 02:07:57:505.724 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-13 Thu 02:07:57:505.797 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:07:57:505.819 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: SANDESH: Send FAILED: 1739412477505439 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-13 Thu 02:07:57:505.828 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:07:57:505.844 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:07:57:505.930 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: SANDESH: Send FAILED: 1739412477505515 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:07:57:505.942 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:07:57:505.951 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:07:57:506.022 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: SANDESH: Send FAILED: 1739412477505572 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:07:57:506.035 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:07:57:506.044 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:07:57:506.058 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: SANDESH: Send FAILED: 1739412477505592 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-13 Thu 02:07:57:506.065 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:07:57:506.073 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:07:57:506.134 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: SANDESH: Send FAILED: 1739412477505786 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:07:57:506.146 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011561981696, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:07:57:656.560 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-13 Thu 02:08:27:735.357 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x279fd70 2025-02-13 Thu 02:08:27:735.393 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.409 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.423 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a06b0 2025-02-13 Thu 02:08:27:735.429 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.439 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.459 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a0ff0 2025-02-13 Thu 02:08:27:735.467 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.481 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.493 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a1930 2025-02-13 Thu 02:08:27:735.501 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.514 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.526 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a2270 2025-02-13 Thu 02:08:27:735.534 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.547 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.560 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x279d830 2025-02-13 Thu 02:08:27:735.566 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.576 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.589 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x2799da0 2025-02-13 Thu 02:08:27:735.597 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.615 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.630 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x279e400 2025-02-13 Thu 02:08:27:735.638 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.650 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.659 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x279eb50 2025-02-13 Thu 02:08:27:735.665 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.673 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.681 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x279f470 2025-02-13 Thu 02:08:27:735.687 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.698 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.713 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a2bb0 2025-02-13 Thu 02:08:27:735.718 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.726 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.736 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a3e30 2025-02-13 Thu 02:08:27:735.742 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.750 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.758 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a34f0 2025-02-13 Thu 02:08:27:735.764 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.771 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.781 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a4770 2025-02-13 Thu 02:08:27:735.786 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.794 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:735.803 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a50b0 2025-02-13 Thu 02:08:27:735.808 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:08:27:735.816 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:08:27:737.947 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:49144::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:08:27:737.980 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:49144::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:08:27:737.989 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:49144::10.0.0.12:8086(-1):EvTcpConnected 2025-02-13 Thu 02:08:27:738.274 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:738.290 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:738.357 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:738.375 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:738.394 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:738.408 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:738.430 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:738.443 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:738.452 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:738.494 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:738.504 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:738.539 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:738.566 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:738.582 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:738.598 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:738.627 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:738.637 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:738.675 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:738.720 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:08:27:738.755 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: SANDESH: Send FAILED: 1739412507735207 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-13 Thu 02:08:27:738.772 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:08:27:738.782 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:08:27:738.791 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:08:27:738.823 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: ClientInit 2025-02-13 Thu 02:08:27:738.828 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:738.863 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:738.930 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:738.942 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:738.839 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-2.:contrail-query-engine:0:Database count 1 2025-02-13 Thu 02:08:27:738.992 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:739.122 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:739.143 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:739.158 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:739.172 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:739.181 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:739.195 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:739.204 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:739.229 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:739.146 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:08:27:739.259 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:08:27:739.283 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: SANDESH: Send FAILED: 1739412507738346 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:08:27:739.295 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:08:27:739.305 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:08:27:739.312 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:08:27:739.333 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: SANDESH: Send FAILED: 1739412507738530 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.46:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:08:27:739.348 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:08:27:739.356 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:08:27:739.363 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:08:27:739.242 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:739.405 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:739.415 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:739.440 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:739.456 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:739.487 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:739.513 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:739.549 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:739.595 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:739.379 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: SANDESH: Send FAILED: 1739412507738816 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.46:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:08:27:739.628 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:08:27:739.638 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:08:27:739.645 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:08:27:739.663 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: SANDESH: Send FAILED: 1739412507738976 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:08:27:739.670 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:08:27:739.683 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:08:27:739.690 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:08:27:739.617 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:739.729 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:739.748 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:08:27:739.758 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:08:27:739.774 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:27:739.704 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:08:27:739.865 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:08:27:739.984 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: SANDESH: Send FAILED: 1739412507739131 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 39 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 41, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:08:27:740.001 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:08:27:740.013 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:08:27:740.021 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:08:27:740.041 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: SANDESH: Send FAILED: 1739412507739581 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:08:27:740.048 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:08:27:740.227 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:08:27:740.237 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:08:27:740.294 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: SANDESH: Send FAILED: 1739412507740102 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1739412507738821 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1739412507740012 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 6 ] 2025-02-13 Thu 02:08:27:740.304 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:08:27:740.313 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:08:27:740.324 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:08:27:740.440 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: SANDESH: Send FAILED: 1739412507740147 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:08:27:740.455 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:08:27:746.605 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: OnMessage control in state: ClientInit 2025-02-13 Thu 02:08:27:746.702 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-13 Thu 02:08:27:746.744 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-13 Thu 02:08:27:746.779 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: Received Ctrl Message with size 0 2025-02-13 Thu 02:08:27:746.952 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1739412507738821 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1739412507740012 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 6 proxy seq 7 2025-02-13 Thu 02:08:27:746.993 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:08:27:747.031 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-02-13 Thu 02:08:27:747.049 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:08:27:747.061 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:08:27:747.072 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:08:27:747.118 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: Established 2025-02-13 Thu 02:08:57:740.435 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-13 Thu 02:09:57:741.111 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-13 Thu 02:10:47:836.269 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x279f470 2025-02-13 Thu 02:10:47:836.330 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:836.414 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x279eb50 2025-02-13 Thu 02:10:47:836.428 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:836.450 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x279e400 2025-02-13 Thu 02:10:47:836.461 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:836.484 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-13 Thu 02:10:47:836.491 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x279d830 2025-02-13 Thu 02:10:47:836.500 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:836.518 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x2799da0 2025-02-13 Thu 02:10:47:836.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:837.037 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:10:47:837.107 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:10:47:837.124 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:10:47:837.140 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:10:47:837.153 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:10:47:851.086 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x27a2270 2025-02-13 Thu 02:10:47:851.550 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:851.633 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x27a1930 2025-02-13 Thu 02:10:47:851.677 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:851.709 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x27a0ff0 2025-02-13 Thu 02:10:47:851.719 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:851.762 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x27a06b0 2025-02-13 Thu 02:10:47:851.804 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:851.838 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-13 Thu 02:10:47:851.845 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x279fd70 2025-02-13 Thu 02:10:47:851.853 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:852.181 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:10:47:852.213 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:10:47:852.231 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:10:47:852.248 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:10:47:852.266 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:10:47:892.326 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x27a50b0 2025-02-13 Thu 02:10:47:892.396 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:892.499 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x27a3e30 2025-02-13 Thu 02:10:47:892.515 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:892.541 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x27a4770 2025-02-13 Thu 02:10:47:892.551 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:892.572 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x27a34f0 2025-02-13 Thu 02:10:47:892.584 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:892.837 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-13 Thu 02:10:47:892.849 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Disconnect status: -1 0x27a2bb0 2025-02-13 Thu 02:10:47:892.858 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:893.020 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:10:47:893.148 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:10:47:893.203 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:10:47:893.217 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:10:47:893.291 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:10:47:899.635 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:49144::10.0.0.12:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-13 Thu 02:10:47:899.667 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:49144::10.0.0.12:8086(-1):EvTcpClose 2025-02-13 Thu 02:10:47:899.733 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvTcpClose in state Established 2025-02-13 Thu 02:10:47:899.747 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Established : EvTcpClose 2025-02-13 Thu 02:10:47:900.092 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Connect : Start Connect timer 10.0.0.46:8086 2025-02-13 Thu 02:10:47:900.209 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:47:900.466 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412647899857 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = cn-jenkins-deploy-platform-ansible-os-2728-3. collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 145389 calls = 19 average_bytes = 7652 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 81, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 14, _iter59->first = sent _iter59->second = 20, ] ] ] 2025-02-13 Thu 02:10:47:900.518 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:47:900.531 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:47:900.555 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412647899941 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:47:900.563 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:47:900.573 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-02-13 Thu 02:10:47:900.614 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:47:900.764 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412647900195 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 145389 calls = 19 average_bytes = 7652 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 81, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 14, _iter59->first = sent _iter59->second = 20, ] ] ] 2025-02-13 Thu 02:10:47:900.782 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:47:902.925 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:46062::10.0.0.46:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:10:47:902.952 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:46062::10.0.0.46:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:10:47:902.963 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:46062::10.0.0.46:8086(-1):EvTcpConnected 2025-02-13 Thu 02:10:47:903.032 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:10:47:903.046 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:10:47:903.081 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: ClientInit 2025-02-13 Thu 02:10:47:903.102 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-2.:contrail-query-engine:0:Database count 2 2025-02-13 Thu 02:10:47:903.191 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:47:903.201 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:10:47:903.247 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: SANDESH: Send FAILED: 1739412647903074 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:47:903.268 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:10:47:903.279 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:47:903.286 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:10:47:903.297 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:47:903.304 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:10:47:903.455 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: SANDESH: Send FAILED: 1739412647903180 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 145389 calls = 19 average_bytes = 7652 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 82, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 20, ] ] ] 2025-02-13 Thu 02:10:47:903.476 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:10:47:929.095 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:46062::10.0.0.46:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-13 Thu 02:10:47:929.131 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:46062::10.0.0.46:8086(-1):EvTcpClose 2025-02-13 Thu 02:10:47:929.215 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-13 Thu 02:10:47:929.246 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: ClientInit : EvTcpClose 2025-02-13 Thu 02:10:47:929.431 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: Return to idle with 5000 2025-02-13 Thu 02:10:47:929.442 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: Idle 2025-02-13 Thu 02:10:47:929.505 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:47:929.659 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: SANDESH: Send FAILED: 1739412647929375 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 146962 calls = 20 average_bytes = 7348 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 82, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 19, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-13 Thu 02:10:47:929.685 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:47:929.696 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:47:929.720 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: SANDESH: Send FAILED: 1739412647929417 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:47:929.728 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:47:929.825 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:10:47:929.864 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:47:930.006 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: SANDESH: Send FAILED: 1739412647929494 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 146962 calls = 20 average_bytes = 7348 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 82, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 19, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-13 Thu 02:10:47:930.025 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:52:837.170 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:10:52:837.378 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:10:52:837.423 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:10:52:837.449 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:10:52:837.479 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:10:52:837.957 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:10:52:838.011 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:10:52:838.028 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:10:52:838.044 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:10:52:838.057 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:10:52:852.259 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:10:52:852.432 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:10:52:852.461 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:10:52:852.492 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:10:52:852.520 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:10:52:852.548 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:10:52:852.581 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:10:52:852.596 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:10:52:852.609 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:10:52:852.620 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:10:52:893.204 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:10:52:893.391 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:10:52:893.435 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:10:52:893.466 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:10:52:893.493 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:10:52:893.908 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:10:52:893.942 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:10:52:893.957 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:10:52:893.971 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:10:52:893.983 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:10:52:929.574 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:10:52:930.175 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Connect : Start Connect timer 10.0.0.35:8086 2025-02-13 Thu 02:10:52:930.297 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:52:930.329 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: SANDESH: Send FAILED: 1739412652929657 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:52:930.376 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:52:930.385 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:52:930.565 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: SANDESH: Send FAILED: 1739412652930282 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 146962 calls = 20 average_bytes = 7348 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 82, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-13 Thu 02:10:52:930.586 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011005511424, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:52:932.783 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:47394::10.0.0.35:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:10:52:932.822 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:47394::10.0.0.35:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:10:52:932.832 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:47394::10.0.0.35:8086(-1):EvTcpConnected 2025-02-13 Thu 02:10:52:932.897 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:10:52:932.923 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:10:52:932.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: ClientInit 2025-02-13 Thu 02:10:52:932.984 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-2.:contrail-query-engine:0:Database count 3 2025-02-13 Thu 02:10:52:933.093 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:52:933.103 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:10:52:933.126 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: SANDESH: Send FAILED: 1739412652932956 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:52:933.134 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:10:52:933.144 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:52:933.151 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:10:52:933.161 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:52:933.168 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:10:52:933.381 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: SANDESH: Send FAILED: 1739412652933079 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 146962 calls = 20 average_bytes = 7348 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-13 Thu 02:10:52:933.414 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:10:54:933.300 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: 10.0.0.35:8086 EvConnectTimerExpired in state ClientInit 2025-02-13 Thu 02:10:54:933.393 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-02-13 Thu 02:10:54:933.404 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: ClientInit : EvConnectTimerExpired 2025-02-13 Thu 02:10:54:933.603 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Return to idle with 5000 2025-02-13 Thu 02:10:54:933.615 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Idle 2025-02-13 Thu 02:10:54:933.677 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:54:933.840 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412654933488 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 148535 calls = 21 average_bytes = 7073 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 26, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-13 Thu 02:10:54:933.872 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:54:933.883 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:54:933.906 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412654933524 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:54:933.913 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:54:933.922 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:10:54:933.931 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:54:934.097 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412654933666 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 148535 calls = 21 average_bytes = 7073 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 26, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-13 Thu 02:10:54:934.116 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:57:741.744 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-13 Thu 02:10:57:741.862 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:57:741.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412657741586 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 51 dequeues = 51 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 64 dequeues = 64 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412654933614 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412654933930 ] max_sm_queue_count = 6 ] 2025-02-13 Thu 02:10:57:741.988 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:57:742.006 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:57:742.247 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412657741722 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 148535 calls = 21 average_bytes = 7073 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 29, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-13 Thu 02:10:57:742.293 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:57:838.050 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:10:57:838.219 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:10:57:838.263 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:10:57:838.304 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:10:57:838.345 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:10:57:838.725 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:10:57:838.784 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:10:57:838.805 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:10:57:838.819 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:10:57:838.837 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:10:57:852.505 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:10:57:852.688 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:10:57:852.736 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:10:57:852.785 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:10:57:852.829 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:10:57:852.857 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:10:57:852.894 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:10:57:852.911 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:10:57:852.923 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:10:57:852.937 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:10:57:893.997 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:10:57:894.198 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:10:57:894.270 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:10:57:894.318 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:10:57:894.382 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:10:57:894.612 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:10:57:894.684 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:10:57:894.701 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:10:57:894.715 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:10:57:894.729 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:10:59:933.760 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:10:59:933.998 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-13 Thu 02:10:59:934.102 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:59:934.133 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412659933853 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:59:934.143 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:59:934.169 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:59:934.333 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412659934091 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 148535 calls = 21 average_bytes = 7073 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-13 Thu 02:10:59:934.365 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:59:936.547 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:55730::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:10:59:936.581 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:55730::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:10:59:936.591 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:55730::10.0.0.12:8086(-1):EvTcpConnected 2025-02-13 Thu 02:10:59:936.672 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:10:59:936.700 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:10:59:936.741 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: ClientInit 2025-02-13 Thu 02:10:59:936.767 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-2.:contrail-query-engine:0:Database count 4 2025-02-13 Thu 02:10:59:936.862 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:59:936.875 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:10:59:936.898 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412659936733 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:59:936.907 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:10:59:936.919 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:59:936.926 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:10:59:936.939 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:59:936.945 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:10:59:937.093 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412659936852 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 148535 calls = 21 average_bytes = 7073 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-13 Thu 02:10:59:937.116 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:11:00:950.702 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.35:55730::10.0.0.12:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-13 Thu 02:11:00:950.745 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.35:55730::10.0.0.12:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:11:00:950.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:55730::10.0.0.12:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-13 Thu 02:11:00:950.777 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:55730::10.0.0.12:8086(-1):EvTcpClose 2025-02-13 Thu 02:11:00:950.905 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-13 Thu 02:11:00:950.948 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: ClientInit : EvTcpClose 2025-02-13 Thu 02:11:00:951.112 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Return to idle with 5000 2025-02-13 Thu 02:11:00:951.124 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Idle 2025-02-13 Thu 02:11:00:951.185 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:00:951.343 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: SANDESH: Send FAILED: 1739412660951058 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:00:951.370 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:00:951.381 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:00:951.404 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: SANDESH: Send FAILED: 1739412660951096 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:00:951.412 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:00:951.421 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:11:00:951.448 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:00:951.589 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: SANDESH: Send FAILED: 1739412660951174 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:00:951.608 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:02:838.831 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:02:838.993 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:02:839.039 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:02:839.072 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:02:839.099 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:02:839.427 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:02:839.456 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:02:839.495 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:02:839.566 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:02:839.582 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:02:852.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:02:852.954 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:02:852.983 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:02:853.019 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:02:853.049 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:02:853.088 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:02:853.104 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:02:853.164 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:02:853.187 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:02:853.234 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:02:894.707 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:02:894.871 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:02:894.909 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:02:894.938 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:02:894.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:02:895.226 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:02:895.377 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:02:895.393 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:02:895.408 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:02:897.051 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:05:951.299 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:11:05:951.588 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Connect : Start Connect timer 10.0.0.46:8086 2025-02-13 Thu 02:11:05:951.710 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:05:951.741 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: SANDESH: Send FAILED: 1739412665951399 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:05:951.750 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:05:951.759 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:05:951.908 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: SANDESH: Send FAILED: 1739412665951698 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:05:951.930 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:05:952.390 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 02:11:05:952.400 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 02:11:05:952.821 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 02:11:05:952.863 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 02:11:05:953.007 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:11:05:953.044 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Idle 2025-02-13 Thu 02:11:05:953.114 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:05:953.267 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: SANDESH: Send FAILED: 1739412665952949 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:05:953.290 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:05:953.301 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:05:953.323 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: SANDESH: Send FAILED: 1739412665952980 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:05:953.331 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:05:953.357 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:11:05:953.379 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:05:953.515 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: SANDESH: Send FAILED: 1739412665953100 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:05:953.533 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:07:839.604 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:07:839.819 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:07:839.873 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:07:839.918 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:07:839.954 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:07:840.264 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:07:840.355 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:07:840.373 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:07:840.389 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:07:840.402 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:07:853.048 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:07:853.245 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:07:854.465 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:07:854.557 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:07:854.594 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:07:854.639 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:07:854.658 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:07:854.672 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:07:854.683 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:07:854.714 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:07:895.423 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:07:895.592 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:07:895.636 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:07:895.662 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:07:896.056 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:07:896.152 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:07:896.172 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:07:896.186 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:07:897.123 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:07:897.626 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:12:840.443 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:12:847.161 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:12:847.492 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:12:847.569 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:12:847.621 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:12:847.685 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:12:847.785 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:12:847.830 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:12:847.966 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:12:848.002 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:12:854.492 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:12:854.693 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:12:854.730 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:12:854.789 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:12:854.806 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:12:854.837 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:12:854.868 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:12:854.900 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:12:854.916 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:12:854.928 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:12:896.186 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:12:896.376 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:12:896.414 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:12:896.444 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:12:896.815 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:12:896.880 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:12:896.905 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:12:896.923 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:12:897.691 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:12:898.091 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:13:953.183 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:11:13:971.091 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Connect : Start Connect timer 10.0.0.35:8086 2025-02-13 Thu 02:11:13:971.093 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 02:11:13:971.191 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 02:11:13:971.305 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:13:971.362 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: SANDESH: Send FAILED: 1739412673970816 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:13:971.376 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:13:971.391 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 02:11:13:971.403 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 02:11:13:971.554 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:11:13:971.599 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Idle 2025-02-13 Thu 02:11:13:971.701 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:13:971.969 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: SANDESH: Send FAILED: 1739412673971275 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:13:972.000 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:13:972.018 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:13:972.247 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: SANDESH: Send FAILED: 1739412673971483 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:13:972.274 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:13:972.292 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:13:972.360 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: SANDESH: Send FAILED: 1739412673971522 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:13:972.376 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:13:972.391 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:11:13:972.422 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:13:972.653 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: SANDESH: Send FAILED: 1739412673971682 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:13:972.689 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:17:847.810 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:17:848.025 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:17:848.068 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:17:848.102 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:17:848.130 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:17:848.546 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:17:848.605 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:17:848.629 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:17:848.644 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:17:849.905 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:17:854.839 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:17:854.982 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:17:855.060 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:17:855.092 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:17:855.109 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:17:855.155 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:17:855.197 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:17:855.242 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:17:855.258 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:17:855.279 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:17:896.911 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:17:897.066 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:17:897.106 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:17:897.135 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:17:897.454 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:17:897.516 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:17:897.533 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:17:897.546 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:17:898.138 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:17:898.526 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:22:848.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:22:848.970 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:22:849.010 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:22:849.035 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:22:849.715 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:22:849.766 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:22:849.782 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:22:849.796 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:22:849.955 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:22:850.491 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:22:855.154 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:22:855.313 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:22:855.360 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:22:855.418 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:22:855.452 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:22:855.729 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:22:855.776 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:22:855.794 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:22:855.808 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:22:855.820 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:22:897.585 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:22:897.774 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:22:897.809 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:22:897.844 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:22:898.249 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:22:898.294 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:22:898.312 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:22:898.592 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:22:898.948 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:22:898.999 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:27:742.489 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:27:742.658 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412687742001 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 71 dequeues = 71 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 94 dequeues = 94 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412673971597 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412673972420 ] max_sm_queue_count = 6 ] 2025-02-13 Thu 02:11:27:742.677 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:27:742.689 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:27:742.860 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SANDESH: Send FAILED: 1739412687742116 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:27:742.896 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:27:849.819 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:27:850.002 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:27:850.043 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:27:850.074 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:27:850.464 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:27:850.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:27:850.547 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:27:850.567 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:27:850.578 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:27:850.861 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:27:855.391 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:27:855.544 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:27:856.222 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:27:856.280 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:27:856.315 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:27:856.367 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:27:856.431 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:27:856.459 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:27:856.475 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:27:856.488 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:27:898.346 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:27:898.559 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:27:898.600 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:27:899.011 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:27:899.051 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:27:899.063 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:27:899.107 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:27:899.120 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:27:899.354 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:27:899.393 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:29:971.747 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:11:29:972.044 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-13 Thu 02:11:29:972.154 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:29:972.184 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: SANDESH: Send FAILED: 1739412689971849 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:29:972.202 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:29:972.211 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:29:972.377 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: SANDESH: Send FAILED: 1739412689972138 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:29:972.393 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:29:972.479 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 02:11:29:972.512 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 02:11:29:972.575 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 02:11:29:972.748 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 02:11:29:972.875 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:11:29:972.925 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: Idle 2025-02-13 Thu 02:11:29:972.995 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:29:973.154 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: SANDESH: Send FAILED: 1739412689972826 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:29:973.172 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:29:973.183 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:29:973.206 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: SANDESH: Send FAILED: 1739412689972854 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:29:973.217 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:29:973.226 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:11:29:973.247 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:29:973.411 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: SANDESH: Send FAILED: 1739412689972979 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:29:973.425 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011510945536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:32:738.536 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:32:738.658 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: SANDESH: Send FAILED: 1739412692738482 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-13 Thu 02:11:32:738.691 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-13 Thu 02:11:32:850.624 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:32:850.774 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:32:850.807 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:32:850.838 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:32:850.893 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:32:851.213 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:32:851.273 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:32:851.293 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:32:851.308 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:32:851.322 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:32:855.630 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:32:855.806 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:32:856.487 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:32:856.575 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:32:856.635 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:32:856.658 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:32:856.669 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:32:856.707 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:32:856.750 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:32:856.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:32:899.090 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:32:899.262 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:32:899.294 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:32:899.386 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:32:899.452 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:32:899.728 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:32:899.837 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:32:899.858 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:32:899.886 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:32:899.903 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:37:851.303 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:37:851.488 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:37:851.527 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:37:851.568 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:37:851.596 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:37:851.999 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:37:852.067 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:37:852.085 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:37:852.098 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:37:852.111 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:37:855.877 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:37:856.079 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:37:856.669 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:37:856.723 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:37:856.738 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:37:856.785 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:37:856.797 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:37:856.831 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:37:856.842 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:37:856.872 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:37:899.871 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:37:900.034 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:37:900.072 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:37:900.105 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:37:900.188 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:37:900.456 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:37:900.594 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:37:900.616 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:37:900.632 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:37:900.647 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:42:852.118 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:42:852.272 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:42:852.311 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:42:852.355 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:42:852.385 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:42:852.718 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:42:852.793 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:42:852.810 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:42:852.826 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:42:852.841 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:42:856.134 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:42:856.262 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:42:856.848 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:42:856.904 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:42:856.935 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:42:856.974 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:42:857.003 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:42:857.016 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:42:857.027 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:42:857.058 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:42:900.548 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:42:900.702 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:42:900.742 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:42:900.767 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:42:900.793 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:42:901.201 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:42:901.266 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:42:901.289 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:42:901.309 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:42:901.324 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:47:852.858 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:47:853.040 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:47:853.081 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:47:853.112 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:47:853.140 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:47:853.617 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:47:853.681 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:47:853.698 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:47:853.712 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:47:853.984 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:47:856.353 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:47:856.506 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:47:857.018 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:47:857.084 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:47:857.131 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:47:857.155 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:47:857.169 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:47:857.205 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:47:857.235 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:47:857.249 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:47:901.301 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:47:901.500 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:47:901.542 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:47:901.571 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:47:901.599 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:47:901.987 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:47:902.049 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:47:902.068 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:47:902.089 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:47:902.103 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:52:853.731 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:52:853.895 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:52:853.943 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:52:853.974 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:52:854.013 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:52:854.311 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:52:854.458 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:52:854.497 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:52:854.536 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:52:854.551 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:52:856.553 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:52:856.674 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:52:857.176 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:52:857.265 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:52:857.368 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:52:857.399 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:52:857.417 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:52:857.464 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:52:857.510 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:52:857.534 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:52:902.097 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:52:902.290 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:52:902.345 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:52:902.374 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:52:902.399 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:52:902.937 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:52:903.012 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:52:903.037 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:52:903.051 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:52:903.065 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:57:742.731 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-13 Thu 02:11:57:742.784 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:57:742.851 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SANDESH: Send FAILED: 1739412717742599 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 79 dequeues = 79 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 105 dequeues = 105 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412689972923 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412692738518 ] max_sm_queue_count = 6 ] 2025-02-13 Thu 02:11:57:742.877 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:57:742.886 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:57:743.099 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SANDESH: Send FAILED: 1739412717742713 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:57:743.120 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:57:854.464 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:11:57:854.657 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:11:57:854.702 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:11:57:854.739 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:11:57:854.764 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:11:57:855.071 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:11:57:855.133 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:11:57:855.150 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:11:57:855.163 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:11:57:855.176 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:11:57:856.743 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:11:57:856.859 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:11:57:857.406 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:11:57:857.468 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:11:57:857.485 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:11:57:857.517 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:11:57:857.541 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:11:57:857.581 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:11:57:857.593 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:11:57:857.622 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:11:57:903.063 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:11:57:903.230 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:11:57:903.278 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:11:57:903.348 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:11:57:903.385 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:11:57:903.792 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:11:57:903.864 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:11:57:903.886 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:11:57:903.906 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:11:57:903.926 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:11:59:973.046 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:11:59:973.283 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Connect : Start Connect timer 10.0.0.46:8086 2025-02-13 Thu 02:11:59:973.397 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:59:973.428 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: SANDESH: Send FAILED: 1739412719973137 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:59:973.438 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:59:973.447 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:59:973.583 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: SANDESH: Send FAILED: 1739412719973384 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 58, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:59:973.605 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:59:974.036 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 02:11:59:974.047 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 02:11:59:974.679 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 02:11:59:974.699 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 02:11:59:974.837 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:11:59:974.878 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Idle 2025-02-13 Thu 02:11:59:974.949 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:59:975.121 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SANDESH: Send FAILED: 1739412719974785 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:59:975.146 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:59:975.157 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:59:975.179 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SANDESH: Send FAILED: 1739412719974810 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:59:975.187 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:59:975.195 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:11:59:975.204 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:59:975.343 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SANDESH: Send FAILED: 1739412719974939 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:11:59:975.363 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:02:855.176 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:12:02:855.342 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:12:02:855.390 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:12:02:855.421 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:12:02:855.447 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:12:02:855.743 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:12:02:855.772 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:12:02:855.788 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:12:02:855.801 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:12:02:855.815 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:12:02:856.901 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:12:02:857.008 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:12:02:857.977 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:12:02:858.063 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:12:02:858.117 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:12:02:858.163 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:12:02:858.228 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:12:02:858.253 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:12:02:858.266 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:12:02:858.278 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:12:02:903.892 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:12:02:904.070 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:12:02:904.124 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:12:02:904.167 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:12:02:904.206 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:12:02:904.548 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:12:02:904.599 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:12:02:904.616 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:12:02:904.631 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:12:02:904.644 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:12:07:855.816 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:12:07:855.940 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:12:07:855.984 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:12:07:856.011 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:12:07:856.037 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:12:07:856.303 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:12:07:856.350 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:12:07:856.367 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:12:07:856.380 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:12:07:856.394 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:12:07:857.083 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:12:07:857.171 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:12:07:858.271 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:12:07:858.360 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:12:07:858.386 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:12:07:858.457 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:12:07:858.530 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:12:07:858.565 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:12:07:858.583 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:12:07:858.595 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:12:07:904.636 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:12:07:904.797 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:12:07:904.830 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:12:07:904.856 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:12:07:904.878 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:12:07:905.298 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:12:07:905.346 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:12:07:905.362 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:12:07:905.376 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:12:07:905.417 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:12:12:856.405 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:12:12:856.553 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:12:12:856.598 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:12:12:856.627 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:12:12:856.655 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:12:12:856.931 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:12:12:857.011 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:12:12:857.043 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:12:12:857.057 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:12:12:857.070 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:12:12:857.214 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:12:12:857.283 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:12:12:858.401 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:12:12:858.480 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:12:12:858.595 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:12:12:858.639 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:12:12:858.653 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:12:12:858.681 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:12:12:858.718 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:12:12:858.734 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:12:12:905.403 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:12:12:905.592 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:12:12:905.642 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:12:12:905.683 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:12:12:905.719 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:12:12:906.145 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:12:12:906.208 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:12:12:906.226 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:12:12:906.240 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:12:12:906.258 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:12:17:857.055 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:12:17:857.219 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:12:17:857.258 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:12:17:857.305 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:12:17:857.344 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:12:17:857.385 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:12:17:857.446 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:12:17:857.501 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:12:17:857.532 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:12:17:857.548 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:12:17:857.607 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:12:17:857.643 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:12:17:858.508 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:12:17:858.563 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:12:17:858.664 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:12:17:858.703 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:12:17:858.747 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:12:17:858.789 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:12:17:858.806 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:12:17:858.846 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:12:17:906.267 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:12:17:906.517 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:12:17:906.602 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:12:17:906.652 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:12:17:906.681 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:12:17:906.876 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:12:17:906.909 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:12:17:906.939 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:12:17:906.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:12:17:906.993 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:12:22:857.566 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:12:22:857.753 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:12:22:857.825 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:12:22:857.851 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:12:22:857.882 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:12:22:857.910 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:12:22:857.942 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:12:22:858.205 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:12:22:858.281 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:12:22:858.305 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:12:22:858.319 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:12:22:858.362 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:12:22:858.588 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:12:22:858.644 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:12:22:858.722 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:12:22:858.758 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:12:22:858.810 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:12:22:858.847 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:12:22:858.871 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:12:22:858.902 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:12:22:906.978 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:12:22:907.163 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:12:22:907.200 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:12:22:907.238 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:12:22:907.283 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:12:22:907.607 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:12:22:907.640 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:12:22:907.653 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:12:22:907.665 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:12:22:907.700 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:12:27:742.993 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:27:743.099 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: SANDESH: Send FAILED: 1739412747742864 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 86 dequeues = 86 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 115 dequeues = 115 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412719974876 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412719975203 ] max_sm_queue_count = 6 ] 2025-02-13 Thu 02:12:27:743.112 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:27:743.122 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:27:743.260 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: SANDESH: Send FAILED: 1739412747742969 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:12:27:743.295 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011022305024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:27:857.971 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:12:27:858.146 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2270 2025-02-13 Thu 02:12:27:859.855 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:12:27:859.963 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:12:27:860.012 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:12:27:860.051 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:12:27:860.095 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:12:27:860.133 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:12:27:860.198 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:12:27:860.251 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:12:27:860.318 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:12:27:860.386 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a1930 2025-02-13 Thu 02:12:27:860.443 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279eb50 2025-02-13 Thu 02:12:27:860.483 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a0ff0 2025-02-13 Thu 02:12:27:860.498 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279e400 2025-02-13 Thu 02:12:27:860.512 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x2799da0 2025-02-13 Thu 02:12:27:860.537 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a06b0 2025-02-13 Thu 02:12:27:860.550 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279fd70 2025-02-13 Thu 02:12:27:860.565 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279f470 2025-02-13 Thu 02:12:27:860.578 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x279d830 2025-02-13 Thu 02:12:27:907.678 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:12:27:907.832 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:12:27:907.892 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:12:27:907.935 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:12:27:907.986 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:12:27:908.364 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a2bb0 2025-02-13 Thu 02:12:27:908.413 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a34f0 2025-02-13 Thu 02:12:27:908.430 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a3e30 2025-02-13 Thu 02:12:27:908.444 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a50b0 2025-02-13 Thu 02:12:27:908.458 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: -1 0x27a4770 2025-02-13 Thu 02:12:29:975.211 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:12:29:975.518 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Connect : Start Connect timer 10.0.0.35:8086 2025-02-13 Thu 02:12:29:975.543 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 02:12:29:975.575 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 02:12:29:975.631 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:12:29:975.663 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: SANDESH: Send FAILED: 1739412749975309 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:12:29:975.673 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:29:975.690 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 02:12:29:975.698 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 02:12:29:975.796 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:12:29:975.828 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Idle 2025-02-13 Thu 02:12:29:975.894 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:29:976.050 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: SANDESH: Send FAILED: 1739412749975615 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 65, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:12:29:976.066 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:29:976.076 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:29:976.225 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: SANDESH: Send FAILED: 1739412749975750 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:12:29:976.243 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:29:976.253 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:29:976.274 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: SANDESH: Send FAILED: 1739412749975775 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:12:29:976.281 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:29:976.290 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:12:29:976.314 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:29:976.457 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: SANDESH: Send FAILED: 1739412749975880 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:12:29:976.474 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:32:739.842 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:32:739.984 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: SANDESH: Send FAILED: 1739412752739794 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-13 Thu 02:12:32:739.998 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-13 Thu 02:12:32:858.217 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2270 2025-02-13 Thu 02:12:32:858.440 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a2270 2025-02-13 Thu 02:12:32:858.456 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:858.498 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:858.626 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:858.639 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:858.678 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:860.473 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a1930 2025-02-13 Thu 02:12:32:860.607 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279eb50 2025-02-13 Thu 02:12:32:860.666 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a1930 2025-02-13 Thu 02:12:32:860.674 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:860.692 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:860.701 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a0ff0 2025-02-13 Thu 02:12:32:860.757 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279e400 2025-02-13 Thu 02:12:32:860.789 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2799da0 2025-02-13 Thu 02:12:32:860.816 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a06b0 2025-02-13 Thu 02:12:32:860.859 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279fd70 2025-02-13 Thu 02:12:32:860.905 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279f470 2025-02-13 Thu 02:12:32:860.945 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x279d830 2025-02-13 Thu 02:12:32:861.005 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a0ff0 2025-02-13 Thu 02:12:32:861.016 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:861.030 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:861.039 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a06b0 2025-02-13 Thu 02:12:32:861.045 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:861.053 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:861.072 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x279fd70 2025-02-13 Thu 02:12:32:861.077 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:861.086 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:861.120 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:861.131 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:861.154 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x279eb50 2025-02-13 Thu 02:12:32:861.160 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:861.169 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:861.178 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x279e400 2025-02-13 Thu 02:12:32:861.183 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:861.191 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:861.284 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:861.324 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:32:861.569 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:861.589 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:861.597 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: SANDESH: Send FAILED: 1739412752861248 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.46:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:12:32:861.602 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x2799da0 2025-02-13 Thu 02:12:32:861.631 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:861.611 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:32:861.644 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:861.678 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x279f470 2025-02-13 Thu 02:12:32:861.683 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:861.692 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:861.738 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:861.762 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:861.774 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:861.787 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:861.796 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:861.804 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x279d830 2025-02-13 Thu 02:12:32:861.809 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:861.818 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:861.851 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:861.868 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:862.019 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:862.038 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:862.061 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:862.081 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:862.102 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:862.186 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:32:862.223 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: SANDESH: Send FAILED: 1739412752862136 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:12:32:862.232 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:32:862.354 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:862.375 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:862.385 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:862.397 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:862.406 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:862.439 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:862.454 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:864.846 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:864.871 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:864.892 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:908.462 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a2bb0 2025-02-13 Thu 02:12:32:908.616 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a34f0 2025-02-13 Thu 02:12:32:908.673 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a3e30 2025-02-13 Thu 02:12:32:908.729 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a50b0 2025-02-13 Thu 02:12:32:908.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27a4770 2025-02-13 Thu 02:12:32:909.104 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a2bb0 2025-02-13 Thu 02:12:32:909.136 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:909.167 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:909.185 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a34f0 2025-02-13 Thu 02:12:32:909.194 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:909.207 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:909.234 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a50b0 2025-02-13 Thu 02:12:32:909.240 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:909.249 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:909.257 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a3e30 2025-02-13 Thu 02:12:32:909.263 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:909.273 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:909.282 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: RAC_Connect status: 0 0x27a4770 2025-02-13 Thu 02:12:32:909.289 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:909.300 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:909.631 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:909.666 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:909.718 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:909.781 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:909.831 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:909.948 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:909.978 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:909.995 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:32:910.072 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: SANDESH: Send FAILED: 1739412752909898 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:12:32:910.084 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011018106624, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:32:910.000 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:910.602 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:910.617 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:910.638 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:910.653 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:910.670 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:910.680 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:910.696 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:57:743.286 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011034900224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-13 Thu 02:12:57:743.415 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011548722944, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:57:743.522 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011548722944, Pid 1]: SANDESH: Send FAILED: 1739412777743153 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 97 dequeues = 97 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 129 dequeues = 129 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412749975827 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412752909981 ] max_sm_queue_count = 6 ] 2025-02-13 Thu 02:12:57:743.538 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011548722944, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:57:743.549 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011548722944, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:57:743.754 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011548722944, Pid 1]: SANDESH: Send FAILED: 1739412777743265 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 74, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:12:57:743.785 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011548722944, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:59:975.921 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:12:59:976.145 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-13 Thu 02:12:59:976.268 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:12:59:976.298 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: SANDESH: Send FAILED: 1739412779976008 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:12:59:976.308 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:59:976.318 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:12:59:976.476 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: SANDESH: Send FAILED: 1739412779976254 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 76, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:12:59:976.508 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011026503424, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:59:979.106 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:42748::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:12:59:979.139 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:42748::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:12:59:979.147 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011709851776, Pid 1]: 10.0.0.35:42748::10.0.0.12:8086(-1):EvTcpConnected 2025-02-13 Thu 02:12:59:979.922 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:12:59:979.940 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:12:59:979.979 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: ClientInit 2025-02-13 Thu 02:12:59:979.992 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-2.:contrail-query-engine:0:Database count 5 2025-02-13 Thu 02:12:59:980.082 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:12:59:980.092 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:12:59:980.116 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SANDESH: Send FAILED: 1739412779979971 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:12:59:980.124 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:12:59:980.133 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:12:59:980.151 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:12:59:980.163 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:12:59:980.170 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:12:59:980.352 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SANDESH: Send FAILED: 1739412779980072 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 78, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-13 Thu 02:12:59:980.375 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:12:59:983.427 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: OnMessage control in state: ClientInit 2025-02-13 Thu 02:12:59:983.481 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-13 Thu 02:12:59:983.490 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-13 Thu 02:12:59:983.515 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Received Ctrl Message with size 0 2025-02-13 Thu 02:12:59:983.787 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 150108 calls = 22 average_bytes = 6823 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 97 dequeues = 97 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 129 dequeues = 129 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412749975827 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412752909981 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 14 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 112517 calls = 13 average_bytes = 8655 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 78, _iter59->first = sent _iter59->second = 23, ] ] max_sm_queue_count = 6 proxy seq 56 2025-02-13 Thu 02:12:59:983.826 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:12:59:983.869 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 37 2025-02-13 Thu 02:12:59:983.888 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:12:59:983.941 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 5 2025-02-13 Thu 02:12:59:983.953 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:12:59:983.964 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:12:59:983.996 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Established 2025-02-13 Thu 02:13:09:855.581 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: OnMessage control in state: Established 2025-02-13 Thu 02:13:09:855.650 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-13 Thu 02:13:09:855.661 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Established : EvSandeshMessageRecv 2025-02-13 Thu 02:13:09:855.705 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: Received Ctrl Message with size 3 2025-02-13 Thu 02:13:09:855.720 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 57, total = 0 2025-02-13 Thu 02:13:09:855.730 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 39, total = 0 2025-02-13 Thu 02:13:09:855.741 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 5, total = 0 2025-02-13 Thu 02:13:09:855.749 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011030701824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:13:57:745.670 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011515143936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-13 Thu 02:14:57:747.363 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011536127744, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-13 Thu 02:15:57:748.644 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011013908224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-13 Thu 02:16:57:749.249 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011566180096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-13 Thu 02:17:57:750.426 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011009709824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-13 Thu 02:18:31:732.371 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvStop in state Established 2025-02-13 Thu 02:18:31:732.469 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Idle 2025-02-13 Thu 02:18:31:732.580 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:18:31:732.592 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:18:31:732.739 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: SANDESH: Send FAILED: 1739413111732568 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739412477500094 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 3463 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 471145 calls = 53 average_bytes = 8889 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 34 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 80, _iter59->first = sent _iter59->second = 57, ] ] ] 2025-02-13 Thu 02:18:31:732.763 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140011544524544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:19:01:696.338 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: http-server-port 8091 2025-02-13 Thu 02:19:01:697.491 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Max-tasks 16 2025-02-13 Thu 02:19:01:697.509 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Max-slice 100 2025-02-13 Thu 02:19:01:697.517 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Collectors 10.0.0.46:8086 2025-02-13 Thu 02:19:01:697.522 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Collectors 10.0.0.35:8086 2025-02-13 Thu 02:19:01:697.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Collectors 10.0.0.12:8086 2025-02-13 Thu 02:19:01:697.560 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: ROLE : Generator 2025-02-13 Thu 02:19:01:697.568 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-13 Thu 02:19:01:697.577 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2728-2. 2025-02-13 Thu 02:19:01:697.584 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-13 Thu 02:19:01:697.592 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-13 Thu 02:19:01:697.600 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-13 Thu 02:19:01:697.609 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-13 Thu 02:19:01:698.399 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: HTTP Introspect Init 2025-02-13 Thu 02:19:01:698.573 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-13 Thu 02:19:01:698.648 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Sandesh Http Server Port: 8091 2025-02-13 Thu 02:19:01:698.699 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-13 Thu 02:19:01:698.709 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-13 Thu 02:19:01:698.795 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Idle 2025-02-13 Thu 02:19:01:698.981 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: No Client: 1739413141698898 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739413141698805 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:19:01:699.136 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-13 Thu 02:19:01:699.154 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-13 Thu 02:19:01:699.167 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Cassandra Servers: 10.0.0.46:9042 10.0.0.35:9042 10.0.0.12:9042 2025-02-13 Thu 02:19:01:699.305 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvStart in state Idle 2025-02-13 Thu 02:19:01:699.434 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Disconnect 2025-02-13 Thu 02:19:01:699.519 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-13 Thu 02:19:01:699.751 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Connect : Start Connect timer 10.0.0.46:8086 2025-02-13 Thu 02:19:01:699.824 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:19:01:699.846 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SANDESH: Send FAILED: 1739413141699424 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-13 Thu 02:19:01:699.857 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:19:01:699.865 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:19:01:699.953 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SANDESH: Send FAILED: 1739413141699507 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739413141698805 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:19:01:699.973 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:19:01:699.983 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:19:01:700.051 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SANDESH: Send FAILED: 1739413141699565 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739413141698805 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:19:01:700.064 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:19:01:700.073 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:19:01:700.087 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SANDESH: Send FAILED: 1739413141699583 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-13 Thu 02:19:01:700.094 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:19:01:700.102 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:19:01:700.218 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SANDESH: Send FAILED: 1739413141699815 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739413141698805 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:19:01:700.233 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:19:01:742.843 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-13 Thu 02:19:01:750.148 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:19:01:750.542 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: SANDESH: Send FAILED: 1739413141748343 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-13 Thu 02:19:01:750.566 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:19:01:751.004 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:58030::10.0.0.46:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-02-13 Thu 02:19:01:751.024 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: 10.0.0.35:58030::10.0.0.46:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:19:01:751.032 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: 10.0.0.35:58030::10.0.0.46:8086(-1):EvTcpConnected 2025-02-13 Thu 02:19:01:751.079 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x105ec70 2025-02-13 Thu 02:19:01:751.086 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.105 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.119 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x105f4d0 2025-02-13 Thu 02:19:01:751.127 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.140 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.154 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x105fd50 2025-02-13 Thu 02:19:01:751.161 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.174 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.188 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x1021240 2025-02-13 Thu 02:19:01:751.195 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.209 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.222 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x1021a00 2025-02-13 Thu 02:19:01:751.230 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.242 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.256 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x1022210 2025-02-13 Thu 02:19:01:751.264 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.274 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.284 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x1022a90 2025-02-13 Thu 02:19:01:751.289 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.297 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.306 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x1023310 2025-02-13 Thu 02:19:01:751.312 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.338 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.353 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x10605d0 2025-02-13 Thu 02:19:01:751.361 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.379 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.394 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x1060e50 2025-02-13 Thu 02:19:01:751.400 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.408 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.417 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x10616d0 2025-02-13 Thu 02:19:01:751.422 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.429 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.439 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x10627d0 2025-02-13 Thu 02:19:01:751.444 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.452 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.461 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x1061f50 2025-02-13 Thu 02:19:01:751.466 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.475 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.483 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x1063050 2025-02-13 Thu 02:19:01:751.488 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.496 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.505 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: RAC_Connect status: 0 0x10638d0 2025-02-13 Thu 02:19:01:751.510 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:19:01:751.517 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:19:01:751.763 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:19:01:751.781 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:19:01:751.818 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: ClientInit 2025-02-13 Thu 02:19:01:751.836 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-2.:contrail-query-engine:0:Database count 1 2025-02-13 Thu 02:19:01:751.927 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:19:01:751.937 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:19:01:751.958 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SANDESH: Send FAILED: 1739413141751810 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-13 Thu 02:19:01:751.966 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:19:01:751.974 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:19:01:751.981 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:19:01:752.004 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:19:01:752.105 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:19:01:752.263 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SANDESH: Send FAILED: 1739413141751918 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739413141698805 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:19:01:752.282 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:19:01:757.683 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:757.718 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:757.733 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:757.742 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:757.755 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:757.772 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:757.804 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:757.814 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:757.826 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:757.919 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:759.011 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:19:01:759.041 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:19:01:759.071 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: SANDESH: Send FAILED: 1739413141758029 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:19:01:759.079 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:19:01:759.102 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:759.444 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:19:01:759.482 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:19:01:759.503 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: SANDESH: Send FAILED: 1739413141759395 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.46:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:19:01:759.516 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:19:01:759.533 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:759.552 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:759.567 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:759.582 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:759.684 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:759.696 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:759.710 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:759.719 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:759.732 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:759.740 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:759.752 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:759.760 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:759.773 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:759.782 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:759.794 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:759.803 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:759.820 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:759.828 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:759.841 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:759.849 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:759.862 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:759.871 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:759.888 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:19:01:759.897 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:19:01:760.564 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: OnMessage control in state: ClientInit 2025-02-13 Thu 02:19:01:760.590 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-13 Thu 02:19:01:760.598 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-13 Thu 02:19:01:760.621 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Received Ctrl Message with size 0 2025-02-13 Thu 02:19:01:760.748 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739413141698805 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-13 Thu 02:19:01:760.783 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:19:01:760.816 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.46:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 6 2025-02-13 Thu 02:19:01:760.837 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:19:01:760.848 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:19:01:760.858 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:19:01:760.893 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696391042816, Pid 1]: Established 2025-02-13 Thu 02:19:01:762.277 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:764.904 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:764.989 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:765.006 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:765.021 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:765.048 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:765.204 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:765.222 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:765.235 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:19:01:765.248 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:20:01:699.781 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695855675136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-13 Thu 02:21:01:700.469 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695851476736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-13 Thu 02:22:01:701.373 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695855675136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-13 Thu 02:23:01:701.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695864071936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-13 Thu 02:24:01:702.616 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-13 Thu 02:25:01:703.347 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-13 Thu 02:26:01:704.040 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695864071936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-13 Thu 02:27:01:704.738 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696378447616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-13 Thu 02:28:01:705.501 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695838881536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-13 Thu 02:29:01:706.109 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695864071936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-13 Thu 02:30:01:706.753 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-13 Thu 02:31:01:708.633 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-13 Thu 02:32:01:709.275 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695851476736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-13 Thu 02:33:01:709.933 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696374249216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-13 Thu 02:33:46:107.160 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:8091::10.0.0.35:49390 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:34:01:710.579 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696370050816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-13 Thu 02:34:20:516.833 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:8091::10.0.0.35:56454 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:34:41:853.659 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:8091::10.0.0.35:46230 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:35:00:116.962 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:8091::10.0.0.35:41242 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:35:01:711.215 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695843079936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-13 Thu 02:36:01:711.762 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696370050816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-13 Thu 02:36:24:009.024 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:8091::10.0.0.46:46280 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:36:47:633.443 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.46 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-13 Thu 02:36:47:634.216 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 5f22cde2-e9b3-11ef-82b3-00000a00002e src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-13 Thu 02:36:47:634.275 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QEQueryLog: 5f22cde2-e9b3-11ef-82b3-00000a00002e :0:json query is: { end_time: 1739414207565731 , enqueue_time: 1739414207633000 , query_metadata: {"enqueue_time": 1739414207632882} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "1664be32-9074-4b6e-9354-a3b867117b57", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:47:634.541 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-13 Thu 02:36:47:634.561 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-13 Thu 02:36:47:634.611 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 5f22cde2-e9b3-11ef-82b3-00000a00002e , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-13 Thu 02:36:47:634.693 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QEQueryLog: 5f22cde2-e9b3-11ef-82b3-00000a00002e :0:json query is: { end_time: 1739414207565731 , enqueue_time: 1739414207633000 , query_metadata: {"enqueue_time": 1739414207632882} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "1664be32-9074-4b6e-9354-a3b867117b57", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:47:681.903 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:681.987 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.006 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.031 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.045 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.075 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.088 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.104 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.118 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.133 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.309 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.330 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.344 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.356 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.368 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.380 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.393 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.405 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.449 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.463 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.314 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.515 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.542 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.554 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.569 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:682.581 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:682.553 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695838881536, Pid 1]: [SYS_INFO]: QEQueryLog: 5f22cde2-e9b3-11ef-82b3-00000a00002e ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 1664be32-9074-4b6e-9354-a3b867117b57 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:47:682.839 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695838881536, Pid 1]: [SYS_INFO]: QEQueryLog: 5f22cde2-e9b3-11ef-82b3-00000a00002e ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 1664be32-9074-4b6e-9354-a3b867117b57 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:47:683.073 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695838881536, Pid 1]: [SYS_INFO]: QEQueryLog: 5f22cde2-e9b3-11ef-82b3-00000a00002e ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 1664be32-9074-4b6e-9354-a3b867117b57 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:47:683.297 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695838881536, Pid 1]: [SYS_INFO]: QEQueryLog: 5f22cde2-e9b3-11ef-82b3-00000a00002e ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 1664be32-9074-4b6e-9354-a3b867117b57 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:47:683.525 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695838881536, Pid 1]: [SYS_INFO]: QEQueryLog: 5f22cde2-e9b3-11ef-82b3-00000a00002e ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 1664be32-9074-4b6e-9354-a3b867117b57 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:47:682.594 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:683.762 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:47:683.778 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 5f22cde2-e9b3-11ef-82b3-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:47:683.914 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QEQueryLog: 5f22cde2-e9b3-11ef-82b3-00000a00002e :0:json query is: { end_time: 1739414207565731 , enqueue_time: 1739414207633000 , query_metadata: {"enqueue_time": 1739414207632882} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "1664be32-9074-4b6e-9354-a3b867117b57", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:47:684.199 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-13 Thu 02:36:47:684.340 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-13 Thu 02:36:47:684.423 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-13 Thu 02:36:47:685.129 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: Finished: QID 5f22cde2-e9b3-11ef-82b3-00000a00002e Table ObjectVMTable Time(ms) 50 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-13 Thu 02:36:47:685.165 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695838881536, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2728-2. table = ObjectVMTable query_stats= [ qid = 5f22cde2-e9b3-11ef-82b3-00000a00002e where = [[{"name": "ObjectId", "value": "1664be32-9074-4b6e-9354-a3b867117b57", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 48,0, 43,0, 47,0, chunk_select_time = 0,0, 0,1, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 50 rows = 5 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-13 Thu 02:36:47:685.211 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696386844416, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-13 Thu 02:36:50:707.991 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.46 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-13 Thu 02:36:50:708.806 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 60f7f0e8-e9b3-11ef-a4f5-00000a00002e src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-13 Thu 02:36:50:708.840 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QEQueryLog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e :0:json query is: { end_time: 1739414210693791 , enqueue_time: 1739414210707476 , query_metadata: {"enqueue_time": 1739414210707337} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455:ctest-vn22-67883455", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:50:709.039 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-13 Thu 02:36:50:709.057 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-13 Thu 02:36:50:709.103 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 60f7f0e8-e9b3-11ef-a4f5-00000a00002e , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-13 Thu 02:36:50:709.251 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695830484736, Pid 1]: [SYS_INFO]: QEQueryLog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e :0:json query is: { end_time: 1739414210693791 , enqueue_time: 1739414210707476 , query_metadata: {"enqueue_time": 1739414210707337} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455:ctest-vn22-67883455", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:50:757.492 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:757.555 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:757.565 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:757.574 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:757.585 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:757.596 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:757.610 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:757.622 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:757.635 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:757.647 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:757.675 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:757.720 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:757.730 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:757.739 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:757.749 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: [SYS_INFO]: QEQueryLog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e :0:json query is: { end_time: 1739414210693791 , enqueue_time: 1739414210707476 , query_metadata: {"enqueue_time": 1739414210707337} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455:ctest-vn22-67883455", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:50:757.754 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:758.019 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:758.028 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:758.036 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:758.043 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:758.051 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:759.133 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:759.158 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:759.167 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:759.175 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:759.190 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:759.197 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:759.205 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:759.221 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:759.229 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e UNION between tables of sizes 0 and 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:50:759.254 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:50:759.485 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QEQueryLog: 60f7f0e8-e9b3-11ef-a4f5-00000a00002e ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:50:759.665 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-13 Thu 02:36:50:759.697 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-13 Thu 02:36:50:759.746 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695859873536, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-13 Thu 02:36:50:761.038 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: Finished: QID 60f7f0e8-e9b3-11ef-a4f5-00000a00002e Table ObjectRoutingInstance Time(ms) 52 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-13 Thu 02:36:50:761.092 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-13 Thu 02:36:50:761.166 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2728-2. table = ObjectRoutingInstance query_stats= [ qid = 60f7f0e8-e9b3-11ef-a4f5-00000a00002e where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455:ctest-vn22-67883455", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 49,0, 48,0, 48,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 52 rows = 1 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-13 Thu 02:37:01:712.905 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696382646016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-13 Thu 02:38:01:713.730 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695851476736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-13 Thu 02:39:01:714.959 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695843079936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-13 Thu 02:40:01:715.894 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695843079936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-13 Thu 02:41:01:716.629 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-13 Thu 02:42:01:717.226 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-13 Thu 02:43:01:717.904 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695838881536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-13 Thu 02:43:41:273.835 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:8091::10.0.0.46:41878 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:44:01:718.635 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695855675136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-13 Thu 02:45:01:719.947 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140695847278336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-13 Thu 02:45:33:088.525 UTC cn-jenkins-deploy-platform-ansible-os-2728-2 [Thread 140696538904704, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.35:8091::10.0.0.35:42382 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256