2025-02-06 Thu 00:50:53:490.888 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: http-server-port 8091 2025-02-06 Thu 00:50:53:490.987 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Max-tasks 16 2025-02-06 Thu 00:50:53:490.995 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Max-slice 100 2025-02-06 Thu 00:50:53:491.001 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Collectors 10.0.0.23:8086 2025-02-06 Thu 00:50:53:491.006 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Collectors 10.0.0.41:8086 2025-02-06 Thu 00:50:53:491.011 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Collectors 10.0.0.38:8086 2025-02-06 Thu 00:50:53:491.030 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: ROLE : Generator 2025-02-06 Thu 00:50:53:491.036 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-06 Thu 00:50:53:491.041 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2696-3. 2025-02-06 Thu 00:50:53:491.046 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-06 Thu 00:50:53:491.051 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-06 Thu 00:50:53:491.057 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-06 Thu 00:50:53:491.062 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-06 Thu 00:50:53:491.558 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: HTTP Introspect Init 2025-02-06 Thu 00:50:53:491.653 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-06 Thu 00:50:53:491.705 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Sandesh Http Server Port: 8091 2025-02-06 Thu 00:50:53:491.738 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-06 Thu 00:50:53:491.746 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-06 Thu 00:50:53:491.818 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Idle 2025-02-06 Thu 00:50:53:491.977 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: No Client: 1738803053491895 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:492.090 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-06 Thu 00:50:53:492.103 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-06 Thu 00:50:53:492.114 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Cassandra Servers: 10.0.0.23:9042 10.0.0.41:9042 10.0.0.38:9042 2025-02-06 Thu 00:50:53:492.167 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Processing scm::EvStart in state Idle 2025-02-06 Thu 00:50:53:492.232 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Disconnect 2025-02-06 Thu 00:50:53:492.300 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-06 Thu 00:50:53:492.478 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 00:50:53:492.543 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:492.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: SANDESH: Send FAILED: 1738803053492224 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:50:53:492.573 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:50:53:492.581 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:492.657 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: SANDESH: Send FAILED: 1738803053492290 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:492.673 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:50:53:492.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:492.750 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: SANDESH: Send FAILED: 1738803053492348 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:492.763 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:50:53:492.771 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:492.786 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: SANDESH: Send FAILED: 1738803053492368 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:50:53:492.792 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:50:53:492.800 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:492.862 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: SANDESH: Send FAILED: 1738803053492533 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:492.874 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:50:53:497.335 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:50:53:497.358 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:50:53:497.418 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:50:53:497.439 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322927511296, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:497.481 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322927511296, Pid 1]: SANDESH: Send FAILED: 1738803053497395 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:50:53:497.490 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322927511296, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:50:58:499.623 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:50:58:499.661 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:50:58:499.701 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:03:501.009 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:51:03:501.052 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:03:501.073 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:08:502.342 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:51:08:502.378 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:08:502.399 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:13:548.277 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:13:548.321 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:13:548.341 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:18:573.048 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:18:573.084 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:18:573.105 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:23:589.623 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:23:589.664 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:23:589.685 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:28:623.623 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:28:623.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:28:623.698 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:33:648.149 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:33:648.192 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:33:648.214 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:38:665.217 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:38:665.254 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:38:665.274 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:43:773.430 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:43:773.561 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:43:773.585 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:48:800.439 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:48:800.487 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:53:829.381 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-06 Thu 00:51:58:834.723 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:51:58:834.764 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 00:51:58:834.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 00:51:58:834.813 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fb95f0 2025-02-06 Thu 00:51:58:834.825 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:834.783 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SANDESH: Send FAILED: 1738803118834621 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:51:58:834.842 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:834.846 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:834.861 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fb9ca0 2025-02-06 Thu 00:51:58:834.903 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:834.876 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 00:51:58:834.926 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 00:51:58:834.926 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:834.962 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fba340 2025-02-06 Thu 00:51:58:834.969 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:834.980 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:834.991 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fb31d0 2025-02-06 Thu 00:51:58:834.997 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.008 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.019 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fba870 2025-02-06 Thu 00:51:58:835.025 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.034 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.045 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fbb1a0 2025-02-06 Thu 00:51:58:835.050 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.068 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.069 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 00:51:58:835.082 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fbbae0 2025-02-06 Thu 00:51:58:835.101 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.110 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.123 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Idle 2025-02-06 Thu 00:51:58:835.136 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fbc420 2025-02-06 Thu 00:51:58:835.151 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.163 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.174 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fbcd60 2025-02-06 Thu 00:51:58:835.179 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.189 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.199 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fbd6a0 2025-02-06 Thu 00:51:58:835.204 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.204 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:835.215 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.238 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fbdfe0 2025-02-06 Thu 00:51:58:835.244 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.266 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fbe920 2025-02-06 Thu 00:51:58:835.271 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.282 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.291 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fbf260 2025-02-06 Thu 00:51:58:835.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.312 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.323 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fbfba0 2025-02-06 Thu 00:51:58:835.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.337 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.337 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SANDESH: Send FAILED: 1738803118835018 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 39 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 40, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:58:835.349 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: RAC_Connect status: 0 0x1fc04e0 2025-02-06 Thu 00:51:58:835.379 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:835.370 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:835.394 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:835.400 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:835.421 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SANDESH: Send FAILED: 1738803118835046 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:51:58:835.428 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:835.437 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 00:51:58:835.476 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:835.576 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SANDESH: Send FAILED: 1738803118835194 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 44 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 46, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:58:835.589 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:835.620 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:835.640 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:835.687 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:835.710 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:835.732 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:835.753 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:835.768 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:835.799 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:835.816 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:835.829 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:835.838 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:835.850 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:835.859 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:835.756 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:835.872 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:835.895 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SANDESH: Send FAILED: 1738803118835675 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 00:51:58:835.906 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:835.909 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:835.929 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:835.933 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:835.968 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:835.987 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:835.994 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SANDESH: Send FAILED: 1738803118835675 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738803118835122 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738803118835475 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 00:51:58:836.006 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:835.996 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:836.015 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:836.027 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:836.040 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:836.055 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:836.067 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:836.105 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.130 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.142 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SANDESH: Send FAILED: 1738803118835740 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 56 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 58, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:58:836.155 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.157 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:836.178 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:836.194 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.200 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SANDESH: Send FAILED: 1738803118835792 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.41:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:58:836.208 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:836.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.218 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:836.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.246 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SANDESH: Send FAILED: 1738803118836183 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:58:836.258 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:836.253 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.537 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:836.551 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:836.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:836.573 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:836.585 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:836.594 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:836.606 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:836.615 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:836.627 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:836.636 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:836.653 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.699 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.716 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.737 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.750 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:836.723 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322914916096, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:836.926 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322914916096, Pid 1]: SANDESH: Send FAILED: 1738803118836681 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:58:836.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322914916096, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:59:835.281 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 00:51:59:835.505 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 00:51:59:835.615 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:51:59:835.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: SANDESH: Send FAILED: 1738803119835359 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:59:835.655 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:59:835.664 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:51:59:835.778 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: SANDESH: Send FAILED: 1738803119835602 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:59:835.802 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:59:837.657 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:33072::10.0.0.23:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 00:51:59:837.685 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: 10.0.0.38:33072::10.0.0.23:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 00:51:59:837.693 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: 10.0.0.38:33072::10.0.0.23:8086(-1):EvTcpConnected 2025-02-06 Thu 00:51:59:837.738 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 00:51:59:837.751 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 00:51:59:837.777 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: ClientInit 2025-02-06 Thu 00:51:59:837.794 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-3.:contrail-query-engine:0:Database count 1 2025-02-06 Thu 00:51:59:837.858 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 00:51:59:837.866 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 00:51:59:837.887 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: SANDESH: Send FAILED: 1738803119837771 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:59:837.894 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 00:51:59:837.902 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 00:51:59:837.908 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 00:51:59:837.918 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 00:51:59:837.924 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 00:51:59:838.033 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: SANDESH: Send FAILED: 1738803119837849 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:59:838.053 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 00:51:59:839.335 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 00:51:59:839.387 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 00:51:59:839.405 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 00:51:59:839.432 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 00:51:59:839.576 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738803118835122 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738803118835475 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2025-02-06 Thu 00:51:59:839.597 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 00:51:59:839.619 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-02-06 Thu 00:51:59:839.630 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 00:51:59:839.637 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-06 Thu 00:51:59:839.645 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 00:51:59:839.672 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: Established 2025-02-06 Thu 00:52:11:475.514 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: OnMessage control in state: Established 2025-02-06 Thu 00:52:11:475.579 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-06 Thu 00:52:11:475.590 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: Established : EvSandeshMessageRecv 2025-02-06 Thu 00:52:11:475.634 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: Received Ctrl Message with size 3 2025-02-06 Thu 00:52:11:475.650 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-02-06 Thu 00:52:11:475.660 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-02-06 Thu 00:52:11:475.673 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-02-06 Thu 00:52:11:475.680 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 00:52:28:836.034 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322910717696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-06 Thu 00:53:28:836.534 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-06 Thu 00:54:28:837.061 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-06 Thu 00:54:52:647.980 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:55992 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-06 Thu 00:55:18:281.216 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323179436160, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:50432 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-06 Thu 00:55:28:838.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-06 Thu 00:56:28:839.279 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-06 Thu 00:57:28:840.065 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-06 Thu 00:58:28:841.211 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322919114496, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-06 Thu 00:59:28:842.316 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-06 Thu 01:00:28:843.865 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-06 Thu 01:01:28:845.298 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322956900096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-06 Thu 01:02:28:845.837 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322927511296, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-06 Thu 01:03:28:846.927 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-06 Thu 01:04:28:848.121 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323031566080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-06 Thu 01:05:28:849.805 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322927511296, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-06 Thu 01:06:28:850.927 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322923312896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-06 Thu 01:07:28:851.492 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322956900096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-06 Thu 01:08:28:852.054 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-06 Thu 01:09:28:854.855 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322923312896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-06 Thu 01:10:28:856.774 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322956900096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-06 Thu 01:11:28:857.533 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322923312896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-06 Thu 01:12:28:859.699 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322956900096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-06 Thu 01:13:28:861.207 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-06 Thu 01:14:28:861.762 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-06 Thu 01:15:28:862.689 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-06 Thu 01:16:28:864.222 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322914916096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-06 Thu 01:17:28:865.237 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322956900096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-06 Thu 01:18:28:865.752 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-06 Thu 01:19:28:866.851 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323031566080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-02-06 Thu 01:20:28:867.401 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-02-06 Thu 01:21:28:867.946 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322923312896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-02-06 Thu 01:22:28:868.891 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323031566080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-02-06 Thu 01:23:28:870.036 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-02-06 Thu 01:24:28:870.915 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-02-06 Thu 01:25:28:871.850 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322914916096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-02-06 Thu 01:26:28:874.783 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322910717696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-02-06 Thu 01:27:28:875.283 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322935908096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-02-06 Thu 01:28:28:875.781 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-02-06 Thu 01:29:28:876.565 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322914916096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-02-06 Thu 01:30:28:879.144 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-02-06 Thu 01:31:28:879.718 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-02-06 Thu 01:32:28:880.712 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322948503296, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-02-06 Thu 01:33:28:881.235 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322923312896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-02-06 Thu 01:34:28:881.742 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322914916096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-02-06 Thu 01:35:28:882.219 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-02-06 Thu 01:36:28:882.813 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322923312896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-02-06 Thu 01:37:28:883.829 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322944304896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-02-06 Thu 01:38:28:884.351 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322931709696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-02-06 Thu 01:39:28:884.966 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322952701696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-02-06 Thu 01:40:28:885.525 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322919114496, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-02-06 Thu 01:41:28:886.116 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322923312896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-02-06 Thu 01:42:28:887.121 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322944304896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-02-06 Thu 01:43:28:887.705 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322914916096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-02-06 Thu 01:44:28:888.875 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-02-06 Thu 01:45:28:889.811 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140323035764480, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-02-06 Thu 01:46:28:890.312 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-02-06 Thu 01:47:28:890.873 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322940106496, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-02-06 Thu 01:48:28:891.832 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322944304896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-02-06 Thu 01:49:28:892.364 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322944304896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-02-06 Thu 01:50:28:892.858 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322906519296, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-02-06 Thu 01:50:39:617.435 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: Processing scm::EvStop in state Established 2025-02-06 Thu 01:50:39:617.525 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: Idle 2025-02-06 Thu 01:50:39:617.626 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:50:39:617.637 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:50:39:617.826 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: SANDESH: Send FAILED: 1738806639617614 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738803053491828 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 2849623 calls = 298 average_bytes = 9562 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 59 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 236 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 101, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 299, ] ] ] 2025-02-06 Thu 01:50:39:617.853 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140322902320896, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:50:55:422.168 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: http-server-port 8091 2025-02-06 Thu 01:50:55:422.249 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Max-tasks 16 2025-02-06 Thu 01:50:55:422.258 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Max-slice 100 2025-02-06 Thu 01:50:55:422.263 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Collectors 10.0.0.23:8086 2025-02-06 Thu 01:50:55:422.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Collectors 10.0.0.41:8086 2025-02-06 Thu 01:50:55:422.273 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Collectors 10.0.0.38:8086 2025-02-06 Thu 01:50:55:422.292 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: ROLE : Generator 2025-02-06 Thu 01:50:55:422.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-06 Thu 01:50:55:422.303 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2696-3. 2025-02-06 Thu 01:50:55:422.308 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-06 Thu 01:50:55:422.313 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-06 Thu 01:50:55:422.318 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-06 Thu 01:50:55:422.324 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-06 Thu 01:50:55:456.573 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: HTTP Introspect Init 2025-02-06 Thu 01:50:55:456.704 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-06 Thu 01:50:55:456.769 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Sandesh Http Server Port: 8091 2025-02-06 Thu 01:50:55:456.805 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-06 Thu 01:50:55:456.813 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-06 Thu 01:50:55:456.891 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Idle 2025-02-06 Thu 01:50:55:457.047 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: No Client: 1738806655456974 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:50:55:457.171 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-06 Thu 01:50:55:457.187 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-06 Thu 01:50:55:457.200 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Cassandra Servers: 10.0.0.23:9042 10.0.0.41:9042 10.0.0.38:9042 2025-02-06 Thu 01:50:55:457.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvStart in state Idle 2025-02-06 Thu 01:50:55:457.339 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Disconnect 2025-02-06 Thu 01:50:55:457.410 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-06 Thu 01:50:55:457.618 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:50:55:457.683 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:50:55:457.703 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806655457331 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:50:55:457.712 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:50:55:457.737 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:50:55:457.814 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806655457399 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:50:55:457.825 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:50:55:457.835 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:50:55:457.902 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806655457458 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:50:55:457.915 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:50:55:457.924 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:50:55:457.939 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806655457478 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:50:55:457.945 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:50:55:457.952 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:50:55:458.014 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806655457672 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:50:55:458.026 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:50:55:550.746 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-06 Thu 01:51:20:568.836 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431918761728, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:51:20:568.899 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431918761728, Pid 1]: SANDESH: Send FAILED: 1738806680568747 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:51:20:568.910 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431918761728, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:51:20:570.792 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:56034::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:51:20:571.040 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:56034::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:51:20:571.054 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:56034::10.0.0.38:8086(-1):EvTcpConnected 2025-02-06 Thu 01:51:20:571.089 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x25405f0 2025-02-06 Thu 01:51:20:571.095 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.108 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.122 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2540ca0 2025-02-06 Thu 01:51:20:571.127 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.136 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.145 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x253a1d0 2025-02-06 Thu 01:51:20:571.150 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.158 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.167 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2541340 2025-02-06 Thu 01:51:20:571.172 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.189 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2541870 2025-02-06 Thu 01:51:20:571.194 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.202 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.210 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x25421a0 2025-02-06 Thu 01:51:20:571.215 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.223 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.231 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2542ae0 2025-02-06 Thu 01:51:20:571.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.244 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.263 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2543420 2025-02-06 Thu 01:51:20:571.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.276 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.284 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2543d60 2025-02-06 Thu 01:51:20:571.289 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.306 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x25446a0 2025-02-06 Thu 01:51:20:571.311 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.319 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.327 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2544fe0 2025-02-06 Thu 01:51:20:571.332 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.340 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.347 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2545920 2025-02-06 Thu 01:51:20:571.353 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.360 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.369 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2546260 2025-02-06 Thu 01:51:20:571.374 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.382 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.391 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2546ba0 2025-02-06 Thu 01:51:20:571.396 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.412 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x25474e0 2025-02-06 Thu 01:51:20:571.417 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:20:571.428 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:20:571.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:51:20:571.688 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:51:20:571.737 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: ClientInit 2025-02-06 Thu 01:51:20:571.765 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-3.:contrail-query-engine:0:Database count 1 2025-02-06 Thu 01:51:20:571.876 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:20:571.886 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:51:20:571.906 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: SANDESH: Send FAILED: 1738806680571731 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:51:20:571.914 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:51:20:571.922 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:20:571.929 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:51:20:571.939 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:20:571.945 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:51:20:572.057 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: SANDESH: Send FAILED: 1738806680571859 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:51:20:572.072 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431914563328, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:51:20:573.701 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.731 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.754 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.764 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.777 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.786 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.799 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.808 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.821 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.829 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.848 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.856 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.869 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.878 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.890 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.899 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.911 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.920 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.936 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.945 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.959 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.968 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:573.980 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:573.989 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:574.002 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:574.011 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:574.023 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:574.032 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:574.045 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:20:574.054 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:20:574.113 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:574.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:20:575.050 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:51:20:575.087 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806680574090 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:51:20:575.102 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:51:20:575.114 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:20:575.121 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:51:20:575.142 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806680574138 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:51:20:575.149 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:51:20:574.907 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:576.116 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:576.904 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:576.932 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:576.948 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:576.963 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:576.977 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:577.000 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:577.016 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:577.035 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:577.074 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:577.089 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:577.102 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:577.115 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:20:576.134 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:20:577.227 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:51:20:577.249 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806680575204 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:51:20:577.257 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:51:20:577.274 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:20:577.282 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:51:20:577.299 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806680577065 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:51:20:577.306 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:51:20:577.424 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 01:51:20:577.445 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 01:51:20:577.458 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 01:51:20:577.483 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 01:51:20:577.602 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-06 Thu 01:51:20:577.624 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:51:20:577.647 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-02-06 Thu 01:51:20:577.685 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:51:20:578.522 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-06 Thu 01:51:20:578.538 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 01:51:20:578.573 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Established 2025-02-06 Thu 01:51:55:457.519 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-06 Thu 01:52:55:458.174 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431253812992, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-06 Thu 01:53:29:876.453 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x25446a0 2025-02-06 Thu 01:53:29:876.526 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:876.659 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x2543d60 2025-02-06 Thu 01:53:29:876.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:876.701 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x2543420 2025-02-06 Thu 01:53:29:876.731 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:876.758 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x2542ae0 2025-02-06 Thu 01:53:29:876.771 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:876.797 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:53:29:876.807 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x25421a0 2025-02-06 Thu 01:53:29:876.815 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:877.066 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:53:29:877.104 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:53:29:877.119 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:53:29:877.133 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:53:29:877.146 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:53:29:964.898 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x2541870 2025-02-06 Thu 01:53:29:964.971 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:965.017 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x2541340 2025-02-06 Thu 01:53:29:965.029 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:965.051 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x253a1d0 2025-02-06 Thu 01:53:29:965.061 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:965.083 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x2540ca0 2025-02-06 Thu 01:53:29:965.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:965.147 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:53:29:965.156 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x25405f0 2025-02-06 Thu 01:53:29:965.182 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:966.130 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:53:29:966.163 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:53:29:966.179 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:53:29:966.199 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:53:29:966.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:53:30:114.802 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x25474e0 2025-02-06 Thu 01:53:30:114.866 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:114.916 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x2546ba0 2025-02-06 Thu 01:53:30:114.927 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:114.956 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x2546260 2025-02-06 Thu 01:53:30:114.966 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:114.994 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x2545920 2025-02-06 Thu 01:53:30:115.003 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:115.036 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:53:30:115.054 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Disconnect status: -1 0x2544fe0 2025-02-06 Thu 01:53:30:115.064 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:124.742 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:53:30:124.802 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:53:30:124.817 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:53:30:124.830 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:53:30:124.842 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:53:30:147.572 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.38:56034::10.0.0.38:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-06 Thu 01:53:30:147.624 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.38:56034::10.0.0.38:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:53:30:147.643 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:56034::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:30:147.651 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:56034::10.0.0.38:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:30:147.731 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvTcpClose in state Established 2025-02-06 Thu 01:53:30:147.750 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Established : EvTcpClose 2025-02-06 Thu 01:53:30:147.979 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:53:30:148.069 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:30:148.229 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SANDESH: Send FAILED: 1738806810147833 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = cn-jenkins-deploy-platform-ansible-os-2696-3. collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161460 calls = 21 average_bytes = 7688 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-06 Thu 01:53:30:148.248 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:148.258 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:30:148.282 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SANDESH: Send FAILED: 1738806810147868 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:30:148.289 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:30:148.298 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-02-06 Thu 01:53:30:148.330 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:30:148.460 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SANDESH: Send FAILED: 1738806810148056 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161460 calls = 21 average_bytes = 7688 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-06 Thu 01:53:30:148.475 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:154.444 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:40690::10.0.0.23:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:30:154.489 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:40690::10.0.0.23:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:30:154.498 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:40690::10.0.0.23:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:30:155.079 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:30:155.115 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:30:155.146 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit 2025-02-06 Thu 01:53:30:155.185 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-3.:contrail-query-engine:0:Database count 2 2025-02-06 Thu 01:53:30:155.287 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:30:155.320 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:30:155.343 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806810155140 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:30:155.351 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:30:155.360 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:30:155.389 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:30:155.400 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:30:155.407 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:155.602 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806810155277 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161460 calls = 21 average_bytes = 7688 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-06 Thu 01:53:30:155.623 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:156.553 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:40690::10.0.0.23:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:30:156.593 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:40690::10.0.0.23:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:30:156.625 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-06 Thu 01:53:30:156.634 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit : EvTcpClose 2025-02-06 Thu 01:53:30:157.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Return to idle with 5000 2025-02-06 Thu 01:53:30:157.048 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Idle 2025-02-06 Thu 01:53:30:157.127 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:30:157.334 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806810156985 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 163033 calls = 22 average_bytes = 7410 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-06 Thu 01:53:30:157.377 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:157.389 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:30:157.431 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806810157028 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:30:157.441 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:30:157.450 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:30:157.502 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:30:157.678 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806810157118 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 163033 calls = 22 average_bytes = 7410 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-06 Thu 01:53:30:158.152 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:34:877.159 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:53:34:877.322 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:53:34:877.378 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:53:34:877.411 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:53:34:877.445 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:53:34:877.493 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:53:34:877.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:53:34:877.536 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:53:34:877.550 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:53:34:877.562 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:53:34:966.207 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:53:34:966.333 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:53:34:966.380 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:53:34:966.413 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:53:34:966.446 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:53:34:966.494 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:53:34:966.532 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:53:34:966.591 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:53:34:966.606 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:53:34:966.621 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:53:35:124.824 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:53:35:124.956 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:53:35:124.980 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:53:35:125.013 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:53:35:125.043 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:53:35:125.071 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:53:35:125.103 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:53:35:125.118 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:53:35:125.130 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:53:35:125.142 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:53:35:157.288 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:35:157.454 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 01:53:35:157.563 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:35:157.591 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806815157338 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:35:157.608 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:35:157.619 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:35:157.770 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806815157542 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 163033 calls = 22 average_bytes = 7410 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-06 Thu 01:53:35:157.785 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:35:159.387 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:57118::10.0.0.41:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:35:159.411 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:57118::10.0.0.41:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:35:159.419 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:57118::10.0.0.41:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:35:159.468 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:35:159.489 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:35:159.516 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: ClientInit 2025-02-06 Thu 01:53:35:159.534 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-3.:contrail-query-engine:0:Database count 3 2025-02-06 Thu 01:53:35:159.604 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:35:159.613 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:35:159.634 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SANDESH: Send FAILED: 1738806815159511 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:35:159.641 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:35:159.650 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:35:159.656 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:35:159.667 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:35:159.674 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:35:159.817 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SANDESH: Send FAILED: 1738806815159593 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 163033 calls = 22 average_bytes = 7410 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-06 Thu 01:53:35:159.834 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:37:159.670 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: 10.0.0.41:8086 EvConnectTimerExpired in state ClientInit 2025-02-06 Thu 01:53:37:159.771 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-02-06 Thu 01:53:37:159.799 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: ClientInit : EvConnectTimerExpired 2025-02-06 Thu 01:53:37:159.961 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Return to idle with 5000 2025-02-06 Thu 01:53:37:159.972 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Idle 2025-02-06 Thu 01:53:37:160.036 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:37:160.188 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: SANDESH: Send FAILED: 1738806817159881 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 164606 calls = 23 average_bytes = 7156 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-06 Thu 01:53:37:160.211 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:37:160.222 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:37:160.245 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: SANDESH: Send FAILED: 1738806817159912 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:37:160.252 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:37:160.262 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:37:160.272 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:37:160.408 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: SANDESH: Send FAILED: 1738806817160025 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 164606 calls = 23 average_bytes = 7156 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-06 Thu 01:53:37:160.427 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:39:877.565 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:53:39:877.670 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:53:39:877.721 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:53:39:877.762 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:53:39:877.800 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:53:39:877.844 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:53:39:877.868 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:53:39:877.882 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:53:39:877.897 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:53:39:877.909 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:53:39:966.563 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:53:39:966.717 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:53:39:966.755 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:53:39:966.786 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:53:39:966.814 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:53:39:966.932 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:53:39:967.014 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:53:39:967.033 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:53:39:967.049 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:53:39:967.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:53:40:125.022 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:53:40:125.171 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:53:40:125.193 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:53:40:125.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:53:40:125.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:53:40:125.285 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:53:40:125.323 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:53:40:125.339 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:53:40:125.351 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:53:40:125.363 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:53:42:160.088 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:42:160.322 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:53:42:160.468 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:42:160.497 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: SANDESH: Send FAILED: 1738806822160155 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:42:160.506 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:42:160.515 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:42:160.652 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: SANDESH: Send FAILED: 1738806822160451 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 164606 calls = 23 average_bytes = 7156 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-06 Thu 01:53:42:160.673 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:42:162.320 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:33018::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:42:162.351 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:33018::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:42:162.360 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:33018::10.0.0.38:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:42:162.853 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:42:162.879 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:42:162.916 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: ClientInit 2025-02-06 Thu 01:53:42:162.940 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-3.:contrail-query-engine:0:Database count 4 2025-02-06 Thu 01:53:42:163.031 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:42:163.041 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:42:163.064 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806822162909 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:42:163.080 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:42:163.088 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:42:163.095 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:42:163.106 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:42:163.112 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:42:163.246 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806822163020 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 164606 calls = 23 average_bytes = 7156 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 29, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-06 Thu 01:53:42:163.261 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:42:163.848 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:33018::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:42:163.868 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:33018::10.0.0.38:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:42:163.910 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-06 Thu 01:53:42:163.926 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: ClientInit : EvTcpClose 2025-02-06 Thu 01:53:42:164.022 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Return to idle with 5000 2025-02-06 Thu 01:53:42:164.031 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Idle 2025-02-06 Thu 01:53:42:164.086 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:42:164.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806822163989 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:42:164.242 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:42:164.252 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:42:164.273 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806822164012 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:42:164.283 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:42:164.292 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:42:164.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:42:164.442 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: SANDESH: Send FAILED: 1738806822164078 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:42:164.455 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431241217792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:44:877.914 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:53:44:878.021 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:53:44:878.069 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:53:44:878.098 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:53:44:878.129 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:53:44:878.176 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:53:44:878.199 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:53:44:878.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:53:44:878.228 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:53:44:878.246 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:53:44:967.017 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:53:44:967.113 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:53:44:967.151 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:53:44:967.193 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:53:44:967.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:53:44:967.321 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:53:44:967.345 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:53:44:967.361 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:53:44:967.374 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:53:44:967.388 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:53:45:125.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:53:45:125.434 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:53:45:125.464 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:53:45:125.500 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:53:45:125.532 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:53:45:125.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:53:45:125.612 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:53:45:125.628 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:53:45:125.641 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:53:45:125.653 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:53:47:164.148 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:47:164.348 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:53:47:164.451 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:47:164.480 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806827164214 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:47:164.489 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:47:164.499 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:47:164.656 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806827164439 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 34, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:47:164.683 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:47:164.660 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:53:47:164.725 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:53:47:164.766 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:53:47:164.776 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:53:47:164.866 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:53:47:164.895 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Idle 2025-02-06 Thu 01:53:47:164.952 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:47:165.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806827164829 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:47:165.113 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:47:165.122 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:47:165.144 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806827164849 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:47:165.151 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:47:165.159 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:47:165.188 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:47:165.324 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806827164943 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:47:165.343 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:49:878.266 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:53:49:878.417 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:53:49:878.452 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:53:49:878.486 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:53:49:878.548 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:53:49:878.618 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:53:49:878.647 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:53:49:878.660 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:53:49:878.696 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:53:49:878.718 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:53:49:967.366 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:53:49:969.368 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:53:49:970.221 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:53:49:970.254 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:53:49:970.282 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:53:49:970.320 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:53:49:970.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:53:49:970.420 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:53:49:970.439 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:53:49:970.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:53:50:125.475 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:53:50:126.830 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:53:50:126.853 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:53:50:126.886 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:53:50:126.916 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:53:50:126.944 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:53:50:126.976 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:53:50:126.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:53:50:127.004 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:53:50:127.016 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:53:54:879.159 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:53:54:879.292 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:53:54:879.334 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:53:54:879.386 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:53:54:879.415 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:53:54:879.471 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:53:54:879.504 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:53:54:879.518 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:53:54:879.530 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:53:54:879.543 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:53:54:970.360 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:53:54:970.436 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:53:54:970.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:53:54:970.506 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:53:54:970.536 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:53:54:970.622 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:53:54:970.650 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:53:54:970.690 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:53:54:970.712 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:53:54:970.747 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:53:55:126.872 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:53:55:126.940 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:53:55:127.003 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:53:55:127.044 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:53:55:127.057 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:53:55:127.085 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:53:55:127.112 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:53:55:127.143 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:53:55:127.158 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:53:55:127.170 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:53:55:165.019 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:55:165.217 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 01:53:55:165.308 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:55:165.336 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806835165095 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:55:165.345 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:55:165.369 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:55:165.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806835165297 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:55:165.523 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:55:165.550 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:53:55:165.571 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:53:55:165.624 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:53:55:165.639 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:53:55:165.744 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:53:55:165.778 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Idle 2025-02-06 Thu 01:53:55:165.837 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:55:165.974 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806835165693 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 41, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:55:165.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:55:166.004 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:55:166.025 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806835165722 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:55:166.033 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:55:166.042 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:55:166.064 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:55:166.191 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806835165828 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 41, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:55:166.204 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:55:459.567 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-06 Thu 01:53:55:459.630 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:55:459.837 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: SANDESH: Send FAILED: 1738806835459447 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 69 dequeues = 69 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 92 dequeues = 92 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806835165776 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806835166062 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:53:55:459.870 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:55:459.880 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:55:460.024 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: SANDESH: Send FAILED: 1738806835459549 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:55:460.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:59:879.535 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:53:59:879.673 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:53:59:879.715 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:53:59:879.757 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:53:59:879.784 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:53:59:879.826 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:53:59:879.856 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:53:59:879.870 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:53:59:879.883 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:53:59:879.897 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:53:59:970.708 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:53:59:970.808 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:53:59:970.846 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:53:59:970.875 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:53:59:970.904 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:53:59:971.002 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:53:59:971.026 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:53:59:971.041 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:53:59:971.055 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:53:59:971.068 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:00:127.019 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:00:127.193 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:00:127.223 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:00:127.257 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:00:127.289 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:00:127.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:00:127.343 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:00:127.356 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:00:127.378 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:00:127.410 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:04:879.907 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:04:880.092 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:04:881.879 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:04:881.919 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:04:881.950 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:04:881.977 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:04:882.027 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:04:882.043 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:04:882.062 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:04:882.075 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:04:971.053 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:04:971.130 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:04:971.166 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:04:971.195 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:04:971.223 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:04:971.294 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:04:971.316 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:04:971.358 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:04:971.381 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:04:971.395 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:05:127.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:05:127.318 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:05:127.348 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:05:127.394 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:05:127.431 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:05:127.462 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:05:127.497 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:05:127.511 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:05:127.522 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:05:127.553 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:09:881.954 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:09:882.116 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:09:882.154 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:09:882.185 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:09:882.232 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:09:882.296 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:09:882.323 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:09:882.344 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:09:882.408 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:09:882.424 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:09:971.355 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:09:971.488 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:09:971.527 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:09:971.556 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:09:971.585 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:09:971.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:09:971.705 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:09:971.733 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:09:971.746 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:09:971.759 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:10:127.391 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:10:127.536 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:10:127.568 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:10:127.602 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:10:127.632 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:10:127.668 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:10:127.683 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:10:127.704 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:10:127.715 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:10:127.746 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:11:165.911 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:54:11:166.193 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:54:11:166.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:54:11:166.304 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:54:11:166.311 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:11:166.346 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806851166004 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:11:166.355 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:11:166.365 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:11:166.516 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806851166287 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:54:11:166.539 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:11:166.549 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:54:11:166.558 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:54:11:166.658 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:54:11:166.724 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Idle 2025-02-06 Thu 01:54:11:166.782 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:11:166.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806851166610 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:54:11:166.954 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:11:166.963 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:11:166.984 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806851166631 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:11:166.991 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:11:166.999 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:54:11:167.018 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:11:167.154 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806851166773 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:54:11:167.173 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:14:882.373 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:14:882.475 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:14:882.518 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:14:882.550 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:14:882.579 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:14:882.629 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:14:882.652 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:14:882.666 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:14:882.711 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:14:882.725 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:14:971.720 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:14:971.817 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:14:971.856 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:14:971.894 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:14:971.924 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:14:972.020 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:14:972.043 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:14:972.058 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:14:972.073 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:14:972.087 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:15:127.576 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:15:127.661 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:15:127.716 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:15:127.751 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:15:127.787 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:15:127.802 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:15:127.812 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:15:127.839 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:15:127.873 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:15:127.888 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:19:882.711 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:19:882.831 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:19:882.875 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:19:882.908 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:19:882.940 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:19:882.989 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:19:883.016 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:19:883.031 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:19:883.046 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:19:883.069 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:19:972.062 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:19:972.147 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:19:972.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:19:972.209 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:19:972.237 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:19:972.293 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:19:972.316 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:19:972.341 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:19:972.356 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:19:972.387 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:20:127.701 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:20:127.801 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:20:127.827 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:20:127.861 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:20:127.904 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:20:127.919 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:20:127.929 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:20:127.955 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:20:127.985 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:20:127.998 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:24:883.057 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:24:883.196 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:24:883.239 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:24:883.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:24:883.309 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:24:883.425 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:24:883.453 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:24:883.468 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:24:883.480 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:24:883.493 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:24:972.338 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:24:972.423 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:24:972.455 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:24:972.484 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:24:972.511 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:24:972.623 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:24:972.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:24:972.661 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:24:972.685 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:24:972.706 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:25:127.837 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:25:127.941 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:25:127.961 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:25:128.002 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:25:128.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:25:128.053 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:25:128.063 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:25:128.091 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:25:128.120 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:25:128.134 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:25:459.839 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:25:459.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806865459703 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 76 dequeues = 76 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 102 dequeues = 102 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806851166722 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806851167017 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:54:25:459.949 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:25:459.959 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:25:460.099 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: SANDESH: Send FAILED: 1738806865459801 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 51, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:54:25:460.120 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431931356928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:25:570.406 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:25:570.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: SANDESH: Send FAILED: 1738806865570365 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-06 Thu 01:54:25:570.483 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-06 Thu 01:54:29:883.482 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:29:883.602 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:29:883.636 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:29:883.667 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:29:883.706 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:29:883.751 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:29:883.773 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:29:883.786 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:29:883.799 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:29:883.814 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:29:972.705 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:29:972.830 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:29:972.871 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:29:972.901 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:29:972.953 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:29:973.079 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:29:973.141 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:29:973.162 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:29:973.178 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:29:973.196 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:30:128.005 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:30:128.139 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:30:128.161 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:30:128.195 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:30:128.233 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:30:128.248 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:30:128.258 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:30:128.285 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:30:128.315 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:30:128.329 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:34:883.809 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:34:883.911 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:34:883.944 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:34:883.972 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:34:883.997 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:34:884.034 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:34:884.057 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:34:884.070 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:34:884.083 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:34:884.095 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:34:973.153 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:34:973.230 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:34:973.261 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:34:973.286 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:34:973.311 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:34:973.393 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:34:973.414 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:34:973.427 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:34:973.453 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:34:973.467 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:35:128.183 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:35:128.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:35:128.326 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:35:128.360 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:35:128.394 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:35:128.410 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:35:128.420 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:35:128.447 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:35:128.478 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:35:128.491 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:39:884.100 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:39:884.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:39:884.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:39:884.325 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:39:884.353 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:39:884.394 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:39:884.436 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:39:884.450 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:39:884.465 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:39:884.478 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:39:973.449 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:39:973.541 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:39:973.576 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:39:973.605 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:39:973.631 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:39:973.807 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:39:973.833 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:39:973.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:39:973.859 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:39:973.872 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:40:128.346 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:40:128.488 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:40:128.511 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:40:128.543 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:40:128.580 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:40:128.595 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:40:128.605 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:40:128.633 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:40:128.663 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:40:128.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:41:166.828 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:54:41:167.012 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:54:41:167.114 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:41:167.143 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SANDESH: Send FAILED: 1738806881166899 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:41:167.151 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:41:167.160 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:41:167.306 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SANDESH: Send FAILED: 1738806881167103 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 54, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:54:41:167.324 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:41:167.352 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:54:41:167.395 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:54:41:167.449 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:54:41:167.477 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:54:41:167.599 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:54:41:167.630 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Idle 2025-02-06 Thu 01:54:41:167.697 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:41:167.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: SANDESH: Send FAILED: 1738806881167548 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:54:41:167.868 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:41:167.878 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:41:167.903 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: SANDESH: Send FAILED: 1738806881167574 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:41:167.909 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:41:167.918 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:54:41:167.940 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:41:168.073 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: SANDESH: Send FAILED: 1738806881167680 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:54:41:168.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431136388864, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:44:884.987 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:44:885.634 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:44:885.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:44:885.748 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:44:885.781 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:44:885.866 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:44:885.908 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:44:885.922 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:44:885.936 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:44:885.948 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:44:973.886 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:44:974.024 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:44:974.064 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:44:974.096 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:44:974.132 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:44:974.211 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:44:974.245 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:44:974.264 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:44:974.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:44:974.323 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:45:128.875 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:45:129.024 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:45:129.063 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:45:129.096 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:45:129.127 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:45:129.182 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:45:129.208 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:45:129.221 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:45:129.233 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:45:129.245 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:49:885.962 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:49:886.571 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:49:886.613 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:49:886.706 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:49:886.749 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:49:886.805 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:49:886.839 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:49:886.853 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:49:886.867 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:49:886.879 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:49:974.280 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:49:974.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:49:974.444 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:49:974.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:49:974.502 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:49:974.578 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:49:974.603 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:49:974.619 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:49:974.634 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:49:974.648 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:50:129.259 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:50:129.400 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:50:129.438 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:50:129.488 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:50:129.512 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:50:129.525 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:50:129.535 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:50:129.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:50:129.595 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:50:129.609 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:54:886.876 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:54:886.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:54:887.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:54:887.071 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:54:887.108 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:54:887.166 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:54:887.198 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:54:887.211 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:54:887.224 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:54:887.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:54:974.633 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:54:974.735 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:54:974.779 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:54:974.808 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:54:974.835 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:54:974.884 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:54:974.927 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:54:974.941 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:54:974.980 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:54:975.003 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:54:55:129.535 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:54:55:129.631 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:54:55:129.651 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:54:55:129.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:54:55:129.721 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:54:55:129.749 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:54:55:129.783 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:54:55:129.798 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:54:55:129.810 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:54:55:129.822 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:54:55:460.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-06 Thu 01:54:55:460.271 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:55:460.477 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: SANDESH: Send FAILED: 1738806895460088 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 113 dequeues = 113 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806881167629 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806881167939 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:54:55:460.503 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:55:460.513 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:55:460.659 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: SANDESH: Send FAILED: 1738806895460195 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 59, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:54:55:460.673 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:59:887.255 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:54:59:887.370 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:54:59:887.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:54:59:887.436 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:54:59:887.465 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:54:59:887.510 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:54:59:887.544 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:54:59:887.559 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:54:59:887.574 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:54:59:887.587 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:54:59:974.955 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:54:59:975.065 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:54:59:975.101 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:54:59:975.129 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:54:59:975.159 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:54:59:975.286 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:54:59:975.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:54:59:975.331 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:54:59:975.345 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:54:59:975.358 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:55:00:129.693 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:55:00:145.493 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:55:00:145.522 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:55:00:145.560 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:55:00:145.588 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:55:00:145.619 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:55:00:145.653 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:55:00:145.668 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:55:00:145.681 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:55:00:145.704 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:55:04:887.576 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:55:04:887.700 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:55:04:887.742 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:55:04:887.770 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:55:04:887.797 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:55:04:887.835 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25446a0 2025-02-06 Thu 01:55:04:887.856 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543d60 2025-02-06 Thu 01:55:04:887.870 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2543420 2025-02-06 Thu 01:55:04:887.883 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2542ae0 2025-02-06 Thu 01:55:04:887.896 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25421a0 2025-02-06 Thu 01:55:04:975.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:55:04:975.402 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:55:04:975.432 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:55:04:975.459 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:55:04:975.486 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:55:04:975.598 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2540ca0 2025-02-06 Thu 01:55:04:975.618 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541340 2025-02-06 Thu 01:55:04:975.631 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2541870 2025-02-06 Thu 01:55:04:975.643 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x253a1d0 2025-02-06 Thu 01:55:04:975.655 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25405f0 2025-02-06 Thu 01:55:05:145.554 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:55:05:145.716 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x25474e0 2025-02-06 Thu 01:55:05:145.740 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:55:05:145.774 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:55:05:145.803 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:55:05:145.839 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546ba0 2025-02-06 Thu 01:55:05:145.854 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2546260 2025-02-06 Thu 01:55:05:145.867 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2545920 2025-02-06 Thu 01:55:05:145.886 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:55:05:145.917 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: -1 0x2544fe0 2025-02-06 Thu 01:55:09:887.902 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25446a0 2025-02-06 Thu 01:55:09:888.031 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543d60 2025-02-06 Thu 01:55:09:888.078 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2543420 2025-02-06 Thu 01:55:09:888.109 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2542ae0 2025-02-06 Thu 01:55:09:888.141 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25421a0 2025-02-06 Thu 01:55:09:888.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x25446a0 2025-02-06 Thu 01:55:09:888.187 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:888.215 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:888.226 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2543d60 2025-02-06 Thu 01:55:09:888.232 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:888.240 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:888.251 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2543420 2025-02-06 Thu 01:55:09:888.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:888.264 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:888.288 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2542ae0 2025-02-06 Thu 01:55:09:888.293 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:888.301 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:888.323 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x25421a0 2025-02-06 Thu 01:55:09:888.329 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:888.340 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:888.424 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:888.448 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:888.483 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:888.497 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:888.506 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:888.517 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:888.526 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:888.593 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:888.612 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:888.624 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:888.634 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:888.644 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431880320768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:09:888.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:888.710 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:888.716 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431880320768, Pid 1]: SANDESH: Send FAILED: 1738806909888558 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:09:888.727 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431880320768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:09:888.730 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:888.765 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:975.656 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2540ca0 2025-02-06 Thu 01:55:09:975.770 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541340 2025-02-06 Thu 01:55:09:975.814 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2541870 2025-02-06 Thu 01:55:09:975.846 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x253a1d0 2025-02-06 Thu 01:55:09:975.876 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25405f0 2025-02-06 Thu 01:55:09:975.931 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2540ca0 2025-02-06 Thu 01:55:09:975.941 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:975.965 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:975.990 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2541340 2025-02-06 Thu 01:55:09:975.996 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:976.008 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:976.035 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2541870 2025-02-06 Thu 01:55:09:976.047 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:976.060 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:976.091 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x253a1d0 2025-02-06 Thu 01:55:09:976.099 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:976.108 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:976.117 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x25405f0 2025-02-06 Thu 01:55:09:976.122 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:976.129 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:976.221 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:976.242 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:976.266 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:976.283 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:976.292 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:976.359 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:976.374 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:976.382 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:976.392 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:09:976.395 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:976.440 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:976.450 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806909976328 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:09:976.459 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:976.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:09:976.481 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:976.495 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:976.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:976.523 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:145.772 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x25474e0 2025-02-06 Thu 01:55:10:145.905 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x25474e0 2025-02-06 Thu 01:55:10:145.915 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:145.948 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:145.958 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546ba0 2025-02-06 Thu 01:55:10:146.007 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2546260 2025-02-06 Thu 01:55:10:146.053 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2545920 2025-02-06 Thu 01:55:10:146.128 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2546ba0 2025-02-06 Thu 01:55:10:146.136 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:146.151 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:146.160 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2546260 2025-02-06 Thu 01:55:10:146.165 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:146.174 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:146.182 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2545920 2025-02-06 Thu 01:55:10:146.187 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:146.194 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:146.201 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2544fe0 2025-02-06 Thu 01:55:10:146.267 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:146.279 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:146.316 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: RAC_Connect status: 0 0x2544fe0 2025-02-06 Thu 01:55:10:146.329 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:146.338 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:146.363 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:146.381 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:146.390 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:146.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:146.412 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:146.424 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:146.433 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:146.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:146.525 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:146.539 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:146.539 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431918761728, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:10:146.579 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431918761728, Pid 1]: SANDESH: Send FAILED: 1738806910146477 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:10:146.587 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431918761728, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:10:146.961 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:146.973 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:146.989 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:11:167.745 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:55:11:167.946 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 01:55:11:168.056 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:11:168.086 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: SANDESH: Send FAILED: 1738806911167821 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:11:168.096 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:11:168.107 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:11:168.205 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:55:11:168.234 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:55:11:168.260 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: SANDESH: Send FAILED: 1738806911168043 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 64, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:55:11:168.282 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:11:168.292 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:55:11:168.300 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:55:11:168.390 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:55:11:168.418 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Idle 2025-02-06 Thu 01:55:11:168.475 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:11:168.612 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: SANDESH: Send FAILED: 1738806911168351 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:55:11:168.629 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:11:168.639 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:11:168.660 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: SANDESH: Send FAILED: 1738806911168370 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:11:168.667 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:11:168.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:55:11:168.736 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:11:168.869 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: SANDESH: Send FAILED: 1738806911168466 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:55:11:168.886 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:25:460.512 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:25:460.667 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806925460353 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 94 dequeues = 94 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 126 dequeues = 126 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806911168417 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806911168734 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:55:25:460.684 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:25:460.702 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:25:460.842 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806925460466 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:55:25:460.863 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:25:571.286 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:25:571.360 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: SANDESH: Send FAILED: 1738806925571233 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-06 Thu 01:55:25:571.371 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431922960128, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-06 Thu 01:55:41:168.554 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:55:41:168.868 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:55:41:168.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:41:169.022 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: SANDESH: Send FAILED: 1738806941168644 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:41:169.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:41:169.047 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:41:169.206 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: SANDESH: Send FAILED: 1738806941168976 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 72, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:55:41:169.223 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:41:173.704 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:33204::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:55:41:173.894 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:33204::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:55:41:173.905 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140432075028608, Pid 1]: 10.0.0.38:33204::10.0.0.38:8086(-1):EvTcpConnected 2025-02-06 Thu 01:55:41:173.981 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:55:41:174.021 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:55:41:174.076 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: ClientInit 2025-02-06 Thu 01:55:41:174.098 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-3.:contrail-query-engine:0:Database count 5 2025-02-06 Thu 01:55:41:174.191 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:55:41:174.200 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:55:41:174.223 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: SANDESH: Send FAILED: 1738806941174064 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:41:174.231 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:55:41:174.240 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:55:41:174.247 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:55:41:174.257 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:55:41:174.263 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:55:41:174.412 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: SANDESH: Send FAILED: 1738806941174180 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 74, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:55:41:174.514 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:55:41:177.066 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 01:55:41:177.111 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 01:55:41:177.121 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 01:55:41:177.153 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 01:55:41:177.378 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 166179 calls = 24 average_bytes = 6924 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 94 dequeues = 94 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 126 dequeues = 126 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806911168417 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806911168734 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 16 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 128587 calls = 15 average_bytes = 8572 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 74, _iter59->first = sent _iter59->second = 25, ] ] max_sm_queue_count = 5 proxy seq 54 2025-02-06 Thu 01:55:41:177.408 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:55:41:177.434 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 37 2025-02-06 Thu 01:55:41:177.445 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:55:41:177.478 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 5 2025-02-06 Thu 01:55:41:177.488 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:55:41:177.499 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 01:55:41:177.528 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: Established 2025-02-06 Thu 01:55:55:460.746 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-06 Thu 01:56:55:462.127 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431245416192, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-06 Thu 01:57:55:463.696 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431262209792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-06 Thu 01:58:55:464.331 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431258011392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-06 Thu 01:59:55:464.917 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431249614592, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-06 Thu 02:00:55:465.765 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431910364928, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-06 Thu 02:01:51:021.770 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431880320768, Pid 1]: Processing scm::EvStop in state Established 2025-02-06 Thu 02:01:51:021.865 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431880320768, Pid 1]: Idle 2025-02-06 Thu 02:01:51:021.978 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431880320768, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 02:01:51:021.989 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431880320768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 02:01:51:022.145 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431880320768, Pid 1]: SANDESH: Send FAILED: 1738807311021964 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1738806655456901 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 513238 calls = 58 average_bytes = 8848 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 38 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 76, _iter59->first = sent _iter59->second = 62, ] ] ] 2025-02-06 Thu 02:01:51:022.169 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140431880320768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 02:02:18:431.285 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: http-server-port 8091 2025-02-06 Thu 02:02:18:431.373 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Max-tasks 16 2025-02-06 Thu 02:02:18:431.381 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Max-slice 100 2025-02-06 Thu 02:02:18:431.387 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Collectors 10.0.0.23:8086 2025-02-06 Thu 02:02:18:431.393 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Collectors 10.0.0.41:8086 2025-02-06 Thu 02:02:18:431.398 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Collectors 10.0.0.38:8086 2025-02-06 Thu 02:02:18:431.423 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: ROLE : Generator 2025-02-06 Thu 02:02:18:431.429 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-06 Thu 02:02:18:431.435 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2696-3. 2025-02-06 Thu 02:02:18:431.440 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-06 Thu 02:02:18:431.445 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-06 Thu 02:02:18:431.450 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-06 Thu 02:02:18:431.456 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-06 Thu 02:02:18:432.050 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: HTTP Introspect Init 2025-02-06 Thu 02:02:18:432.154 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-06 Thu 02:02:18:432.214 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Sandesh Http Server Port: 8091 2025-02-06 Thu 02:02:18:432.252 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-06 Thu 02:02:18:432.261 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-06 Thu 02:02:18:432.340 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Idle 2025-02-06 Thu 02:02:18:432.499 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: No Client: 1738807338432422 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807338432349 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 02:02:18:432.640 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-06 Thu 02:02:18:432.666 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-06 Thu 02:02:18:432.683 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Cassandra Servers: 10.0.0.23:9042 10.0.0.41:9042 10.0.0.38:9042 2025-02-06 Thu 02:02:18:432.826 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Processing scm::EvStart in state Idle 2025-02-06 Thu 02:02:18:432.926 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Disconnect 2025-02-06 Thu 02:02:18:433.007 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-06 Thu 02:02:18:433.245 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 02:02:18:433.319 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:02:18:433.343 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: SANDESH: Send FAILED: 1738807338432915 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 02:02:18:433.353 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 02:02:18:433.362 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:02:18:433.456 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: SANDESH: Send FAILED: 1738807338432995 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807338432349 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 02:02:18:433.468 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 02:02:18:433.477 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:02:18:433.545 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: SANDESH: Send FAILED: 1738807338433055 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807338432349 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 02:02:18:433.559 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 02:02:18:433.568 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:02:18:433.582 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: SANDESH: Send FAILED: 1738807338433073 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 02:02:18:433.588 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 02:02:18:433.595 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:02:18:433.668 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: SANDESH: Send FAILED: 1738807338433310 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807338432349 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 02:02:18:433.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 02:02:18:463.509 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-06 Thu 02:02:18:469.737 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:02:18:469.785 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: SANDESH: Send FAILED: 1738807338468153 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 02:02:18:469.796 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 02:02:18:470.277 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:43424::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-02-06 Thu 02:02:18:470.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: 10.0.0.38:43424::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 02:02:18:470.304 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: 10.0.0.38:43424::10.0.0.38:8086(-1):EvTcpConnected 2025-02-06 Thu 02:02:18:470.327 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x2117240 2025-02-06 Thu 02:02:18:470.334 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.344 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.356 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x2118210 2025-02-06 Thu 02:02:18:470.361 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.369 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.377 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x2117a00 2025-02-06 Thu 02:02:18:470.382 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.389 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.397 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x2118a90 2025-02-06 Thu 02:02:18:470.402 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.410 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.417 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x2154c70 2025-02-06 Thu 02:02:18:470.422 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.430 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.438 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x2119310 2025-02-06 Thu 02:02:18:470.442 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.450 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.457 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x21554d0 2025-02-06 Thu 02:02:18:470.462 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.470 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.481 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x2155d50 2025-02-06 Thu 02:02:18:470.486 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.496 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.504 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x21565d0 2025-02-06 Thu 02:02:18:470.509 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.517 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.525 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x21576d0 2025-02-06 Thu 02:02:18:470.530 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.537 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.545 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x2156e50 2025-02-06 Thu 02:02:18:470.550 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.557 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.565 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x2157f50 2025-02-06 Thu 02:02:18:470.570 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.577 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.585 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x21587d0 2025-02-06 Thu 02:02:18:470.590 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.597 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.605 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x2159050 2025-02-06 Thu 02:02:18:470.610 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.617 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.624 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: RAC_Connect status: 0 0x21598d0 2025-02-06 Thu 02:02:18:470.629 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:02:18:470.637 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:02:18:470.887 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 02:02:18:470.904 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 02:02:18:470.938 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: ClientInit 2025-02-06 Thu 02:02:18:470.958 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-3.:contrail-query-engine:0:Database count 1 2025-02-06 Thu 02:02:18:471.042 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:02:18:471.053 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 02:02:18:471.073 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: SANDESH: Send FAILED: 1738807338470929 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 02:02:18:471.080 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 02:02:18:471.089 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:02:18:471.095 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 02:02:18:471.115 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:02:18:471.121 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 02:02:18:471.238 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: SANDESH: Send FAILED: 1738807338471032 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738807338432349 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 02:02:18:471.254 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 02:02:18:471.928 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:471.945 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:471.964 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:471.973 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:471.985 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:471.994 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.006 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.014 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.026 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.035 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.046 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.055 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.067 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.076 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.088 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.096 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.108 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.117 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.128 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.140 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.153 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.161 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.173 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.181 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.192 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.201 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.212 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.220 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.232 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:02:18:472.240 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:02:18:472.322 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.351 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.372 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.387 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.401 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.415 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.440 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.460 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.485 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.497 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.509 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.533 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.548 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:02:18:472.648 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:02:18:472.667 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 02:02:18:472.689 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: SANDESH: Send FAILED: 1738807338472274 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 02:02:18:472.696 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 02:02:18:472.705 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:02:18:472.711 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 02:02:18:472.729 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: SANDESH: Send FAILED: 1738807338472345 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.41:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 02:02:18:472.740 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 02:02:18:472.748 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:02:18:472.755 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 02:02:18:472.771 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: SANDESH: Send FAILED: 1738807338472367 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 02:02:18:472.777 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 02:02:18:472.786 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:02:18:472.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 02:02:18:472.808 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: SANDESH: Send FAILED: 1738807338472454 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 02:02:18:472.814 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 02:02:18:474.976 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 02:02:18:476.369 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 02:02:18:476.385 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 02:02:18:476.416 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 02:02:18:477.147 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738807338432349 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-06 Thu 02:02:18:477.215 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 02:02:18:477.249 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-02-06 Thu 02:02:18:477.262 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 02:02:18:477.270 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-06 Thu 02:02:18:477.278 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 02:02:18:477.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: Established 2025-02-06 Thu 02:03:18:433.068 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429202773760, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-06 Thu 02:04:18:434.544 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429619504896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-06 Thu 02:05:18:435.217 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429615306496, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-06 Thu 02:06:18:435.863 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-06 Thu 02:07:18:436.947 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429619504896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-06 Thu 02:08:18:437.567 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429611108096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-06 Thu 02:09:18:438.204 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429232162560, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-06 Thu 02:10:18:439.274 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429232162560, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-06 Thu 02:11:18:439.919 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-06 Thu 02:12:18:440.589 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429211170560, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-06 Thu 02:13:18:441.169 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429606909696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-06 Thu 02:14:18:441.744 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429223765760, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-06 Thu 02:15:11:192.563 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:56814 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:15:18:442.347 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429619504896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-06 Thu 02:15:35:109.336 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:42938 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:16:18:442.971 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429227964160, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-06 Thu 02:16:54:622.753 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.23:40538 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:17:12:745.111 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.23 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-06 Thu 02:17:12:745.693 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 79f4d048-e430-11ef-aed3-00000a000017 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-06 Thu 02:17:12:745.738 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QEQueryLog: 79f4d048-e430-11ef-aed3-00000a000017 :0:json query is: { end_time: 1738808232731299 , enqueue_time: 1738808232744638 , query_metadata: {"enqueue_time": 1738808232744524} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941:ctest-vn22-78729941", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:12:745.976 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: Chunks: 2 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-06 Thu 02:17:12:746.011 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-06 Thu 02:17:12:746.073 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 79f4d048-e430-11ef-aed3-00000a000017 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-06 Thu 02:17:12:746.208 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429232162560, Pid 1]: [SYS_INFO]: QEQueryLog: 79f4d048-e430-11ef-aed3-00000a000017 :0:json query is: { end_time: 1738808232731299 , enqueue_time: 1738808232744638 , query_metadata: {"enqueue_time": 1738808232744524} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941:ctest-vn22-78729941", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:12:788.211 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:12:788.251 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:12:788.260 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:12:788.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:12:788.276 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:12:788.284 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:12:788.292 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:12:788.299 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:12:788.306 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:12:788.324 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:12:788.388 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429602711296, Pid 1]: [SYS_INFO]: QEQueryLog: 79f4d048-e430-11ef-aed3-00000a000017 :0:json query is: { end_time: 1738808232731299 , enqueue_time: 1738808232744638 , query_metadata: {"enqueue_time": 1738808232744524} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941:ctest-vn22-78729941", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:12:792.243 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:12:792.267 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:12:792.276 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:12:792.284 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:12:792.291 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 UNION between tables of sizes 0 and 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:12:792.300 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:12:792.309 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:12:792.317 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:12:792.324 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:12:792.339 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: 79f4d048-e430-11ef-aed3-00000a000017 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:12:792.573 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QEQueryLog: 79f4d048-e430-11ef-aed3-00000a000017 ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:12:792.740 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-06 Thu 02:17:12:792.768 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-06 Thu 02:17:12:792.812 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-06 Thu 02:17:12:793.247 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429206972160, Pid 1]: [SYS_INFO]: QELog: Finished: QID 79f4d048-e430-11ef-aed3-00000a000017 Table ObjectRoutingInstance Time(ms) 47 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-06 Thu 02:17:12:793.371 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429198575360, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2696-3. table = ObjectRoutingInstance query_stats= [ qid = 79f4d048-e430-11ef-aed3-00000a000017 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941:ctest-vn22-78729941", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 2 chunk_where_time = 41,0, 44,0, chunk_select_time = 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 47 rows = 1 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-06 Thu 02:17:12:793.491 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429206972160, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-06 Thu 02:17:18:443.783 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-06 Thu 02:18:18:446.417 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429597648640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-06 Thu 02:19:18:447.033 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429623703296, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-06 Thu 02:20:18:447.549 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429223765760, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-06 Thu 02:21:18:448.106 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429615306496, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-06 Thu 02:22:18:448.641 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429597648640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-06 Thu 02:23:18:449.250 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429198575360, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-06 Thu 02:23:32:743.608 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.23:49708 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:24:18:449.834 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429223765760, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-06 Thu 02:25:18:450.381 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429219567360, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-06 Thu 02:25:42:239.159 UTC cn-jenkins-deploy-platform-ansible-os-2696-3 [Thread 140429771565184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:48262 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256