2025-09-11 Thu 04:17:04:916.458 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_INFO]: VersionInfoLog: {"build-info":[{"build-version":"v0+nightly.r24.1.centos","build-time":"2025-09-11 00:09:15.426696","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-10925-1.novalocal","build-id":"v0+nightly.r24.1.centos-nightly_r24.1_centos.el7@contrail","build-number":"unknown"}]} src/contrail-common/base/misc_utils.cc 41 2025-09-11 Thu 04:17:04:916.606 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: ROLE : Generator 2025-09-11 Thu 04:17:04:916.617 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: MODULE : contrail-dns 2025-09-11 Thu 04:17:04:916.623 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-4080-1 2025-09-11 Thu 04:17:04:916.628 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: NODE TYPE : Control 2025-09-11 Thu 04:17:04:916.633 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: INSTANCE ID : 0 2025-09-11 Thu 04:17:04:916.640 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: HTTP SERVER PORT : 8092 2025-09-11 Thu 04:17:04:916.651 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-09-11 Thu 04:17:04:916.910 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: HTTP Introspect Init 2025-09-11 Thu 04:17:04:917.026 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8092 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-11 Thu 04:17:04:917.084 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: Sandesh Http Server Port: 8092 2025-09-11 Thu 04:17:04:917.136 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: NOT Writing http_port 8092TO : /tmp/contrail-dns.77239.http_port 2025-09-11 Thu 04:17:04:917.145 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: CONNECT TO COLLECTOR: 1 2025-09-11 Thu 04:17:04:917.228 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: Idle 2025-09-11 Thu 04:17:04:917.401 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: No Client: 1757564224917326 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:04:917.676 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvStart in state Idle 2025-09-11 Thu 04:17:04:917.725 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Disconnect 2025-09-11 Thu 04:17:04:917.751 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: Logging: DISABLED -> ENABLED 2025-09-11 Thu 04:17:04:917.772 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-09-11 Thu 04:17:04:920.455 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvCollectorUpdate in state Disconnect 2025-09-11 Thu 04:17:04:920.666 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Connect : Start Connect timer 10.0.0.254:8086 2025-09-11 Thu 04:17:04:920.737 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:04:920.831 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: SANDESH: Send FAILED: 1757564224917803 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Disconnect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:04:920.849 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:04:920.859 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:04:920.926 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: SANDESH: Send FAILED: 1757564224920526 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Disconnect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:04:920.936 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:04:920.947 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:04:921.007 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: SANDESH: Send FAILED: 1757564224920728 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:04:921.017 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:05:230.290 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 10.20.0.17:8093 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-11 Thu 04:17:05:230.401 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: Bulk sync has been started controller/src/dns/main.cc 380 2025-09-11 Thu 04:17:05:230.495 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Config Client Mgr SM: Start RabbitMqReader and init Database src/contrail-common/config-client-mgr/config_client_manager.cc 295 2025-09-11 Thu 04:17:05:230.609 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection down src/contrail-common/config-client-mgr/config_amqp_client.cc 159 2025-09-11 Thu 04:17:05:230.618 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:230.655 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564225230584 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Down description = RabbitMQ connection down ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Database:Cassandra ], ] ] ] 2025-09-11 Thu 04:17:05:230.665 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:05:230.622 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: Connect to Rabbit MQ with queue_delete TRUE src/contrail-common/config-client-mgr/config_amqp_client.cc 168 2025-09-11 Thu 04:17:05:398.214 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection established src/contrail-common/config-client-mgr/config_amqp_client.cc 152 2025-09-11 Thu 04:17:05:398.258 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Config Client Mgr SM: Init Database src/contrail-common/config-client-mgr/config_client_manager.cc 300 2025-09-11 Thu 04:17:05:398.302 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Lost Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 578 2025-09-11 Thu 04:17:05:398.313 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db Init src/contrail-common/config-client-mgr/config_cassandra_client.cc 85 2025-09-11 Thu 04:17:05:399.964 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:400.013 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564225398171 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Database:Cassandra ], ] ] ] 2025-09-11 Thu 04:17:05:400.024 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:05:400.042 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:400.061 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564225398296 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Down description = Lost Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector, Database:Cassandra connection down ], ] ] ] 2025-09-11 Thu 04:17:05:400.068 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:05:454.980 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db SetTableSpace src/contrail-common/config-client-mgr/config_cassandra_client.cc 93 2025-09-11 Thu 04:17:05:457.417 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db UseColumnFamily uuidTableName src/contrail-common/config-client-mgr/config_cassandra_client.cc 102 2025-09-11 Thu 04:17:05:457.477 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db UseColumnFamily fqnTableName src/contrail-common/config-client-mgr/config_cassandra_client.cc 108 2025-09-11 Thu 04:17:05:457.704 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:457.722 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: BulkDataSync Started src/contrail-common/config-client-mgr/config_cassandra_client.cc 391 2025-09-11 Thu 04:17:05:457.741 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:457.803 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: SANDESH: Send FAILED: 1757564225457515 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:05:457.824 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:05:461.548 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:475.636 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:486.502 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:489.698 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:496.869 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:502.567 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:503.667 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:508.287 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:510.003 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:513.462 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:515.815 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:517.075 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:518.189 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:521.174 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:524.294 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:525.330 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:526.569 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:529.833 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:531.708 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:538.155 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:543.130 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:553.040 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:554.253 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:566.426 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:569.396 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:574.240 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:580.659 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:582.024 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:586.532 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:588.791 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:591.738 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:592.831 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:602.215 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:606.000 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:607.912 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:613.486 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:614.880 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:618.256 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:625.482 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:626.587 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:633.762 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:634.998 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:636.381 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:651.006 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:655.603 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:666.682 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:672.179 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:678.428 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:687.560 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:694.560 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:708.235 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:710.816 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:714.371 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:715.724 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:718.492 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:731.004 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:734.287 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:738.534 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:750.811 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:752.940 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:754.656 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:762.577 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:764.628 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:772.817 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:774.740 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:777.008 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:789.645 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:791.110 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:792.548 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:793.672 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:798.822 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:804.172 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:807.588 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:812.472 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:813.710 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:815.160 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:822.493 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:833.656 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:835.525 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:842.445 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:854.274 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:855.433 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:882.072 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:885.732 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:887.490 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:900.495 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:902.386 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:905.610 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:934.631 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:967.049 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:972.499 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:003.232 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:005.181 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:006.658 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:010.906 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:013.738 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:016.035 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:044.986 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:047.088 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:049.595 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:061.924 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:069.319 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:073.798 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:083.009 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:086.353 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:092.203 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:105.257 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:110.759 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:138.822 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:279.561 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:287.030 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:291.989 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:303.700 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:337.743 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:347.768 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:363.344 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:374.352 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:379.833 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:385.603 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:386.610 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:387.750 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:389.066 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:391.221 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:393.040 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:393.261 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:395.218 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:395.883 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: BulkSyncDone by all readers src/contrail-common/config-client-mgr/config_cassandra_client.cc 552 2025-09-11 Thu 04:17:06:398.064 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Config Client Mgr SM: End of RIB computed and notification sent src/contrail-common/config-client-mgr/config_client_manager.cc 191 2025-09-11 Thu 04:17:06:398.094 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: TRUE src/contrail-common/config-client-mgr/config_client_manager.cc 219 2025-09-11 Thu 04:17:06:400.223 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-core-files controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:400.550 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating global-system-config:default-global-system-config controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:400.579 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:400.597 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating network-ipam:default-domain:default-project:default-network-ipam controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:400.623 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating project:default-domain:default-project controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:400.636 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-network-ipam controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:400.875 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating policy-management:default-policy-management controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:400.894 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-bottle-request-size-limit controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:400.931 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:400.959 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating domain:default-domain controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:400.973 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-project controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:405.683 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-address-mismatch-compute controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:405.727 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:406.226 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:06:406.266 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: SANDESH: Send FAILED: 1757564226398229 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:06:406.275 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:06:407.285 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:407.686 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating application-policy-set:default-domain:default-project:default-application-policy-set controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:407.714 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-application-policy-set controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:407.730 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-table-limit-exceeded controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:407.749 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:407.773 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:hitless_upgrade_strategy_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:407.804 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:407.820 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:image_upgrade_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:407.839 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:407.851 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating fabric:default-global-system-config:default-fabric controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:407.868 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-fabric controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:407.886 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-appliance-set:default-global-system-config:opencontrail controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:407.903 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-service-appliance-set controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:411.495 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:411.885 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:application controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:411.907 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:411.927 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:411.943 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:dci-network controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:411.975 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:411.989 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating api-access-list:default-global-system-config:default-api-access-list controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:412.004 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-api-access-list controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:412.018 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-conf-incorrect controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:412.035 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:412.052 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating security-group:default-domain:default-project:__no_rule__ controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:412.068 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-security-group controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:412.940 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:413.408 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-partial-sysinfo controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:413.433 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:414.221 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:haproxy-loadbalancer-template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:414.249 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:414.277 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-package-version-mismatch controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:414.295 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:414.308 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-disk-usage-critical controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:414.324 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:414.336 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-prouter-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:414.358 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:414.373 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-conf-elements-inconsistency controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:414.388 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:415.093 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:site controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:415.108 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:415.127 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:415.140 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:415.162 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:415.179 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:415.194 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:417.944 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:418.849 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:cli_sync_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:418.885 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:418.901 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:fabric_config_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:418.921 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:418.934 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:netns-snat-template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:418.953 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:418.967 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:container_cleanup_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:418.987 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.000 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:label controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.014 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-address-mismatch-control controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.031 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.044 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:maintenance_mode_deactivate_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.070 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.083 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.099 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.111 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-xmpp-close-reason controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.129 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.141 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.156 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.168 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:neutron_fwaas controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.179 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:docker-template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.194 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.206 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-bgp-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.223 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.235 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-appliance-set:default-global-system-config:native controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.251 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-service-appliance-set controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.263 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:discover_os_computes_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.290 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.302 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-process-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.334 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:419.348 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:nat-template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:419.363 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:421.508 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:422.369 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-storage-cluster-state controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.401 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.417 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:discover_role_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.439 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.453 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-limit-exceeded controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.469 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.482 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:ztp_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.667 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.685 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-node-status controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.701 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.719 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:ip-fabric controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.750 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.763 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:deployment controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.776 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:analyzer-template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.800 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.813 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-interface controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.829 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.842 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-disk-usage-high controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.856 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.868 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:discover_device_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.887 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.899 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating global-qos-config:default-global-system-config:default-global-qos-config controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.913 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-global-qos-config controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:422.926 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-process-status controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:422.941 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:426.568 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:426.961 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:default-virtual-network controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:426.991 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:427.007 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:neutron_tag controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:427.021 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:427.039 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:427.053 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating application-policy-set:default-policy-management:default-application-policy-set controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:427.070 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating policy-management-application-policy-set controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:427.086 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating global-vrouter-config:default-global-system-config:default-global-vrouter-config controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:427.100 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating application-policy-set-global-vrouter-config controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:427.114 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-pending-cassandra-compaction-tasks controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:427.131 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:427.153 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-global-vrouter-config controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:427.167 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating discovery-service-assignment:default-discovery-service-assignment controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:427.178 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-xmpp-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:427.198 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:428.801 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:429.234 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating network-ipam:default-domain:default-project:service-chain-flat-ipam controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:429.260 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-network-ipam controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:429.275 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:__link_local__ controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:429.304 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:429.317 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-prouter-tsn-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:429.336 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:429.353 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:tier controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:429.366 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:maintenance_mode_activate_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:429.388 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:429.402 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating namespace:default-domain:default-namespace controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:429.417 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-namespace controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:429.430 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:device_import_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:429.449 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:430.727 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: EndOfRib point was reached, IF-MAP objects have been created controller/src/dns/main.cc 86 2025-09-11 Thu 04:17:06:430.756 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.virtual_DNS.0, counting 0 elements controller/src/dns/main.cc 98 2025-09-11 Thu 04:17:06:430.773 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.virtual_DNS_record.0, counting 0 elements controller/src/dns/main.cc 98 2025-09-11 Thu 04:17:06:430.792 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.network_ipam.0, counting 2 elements controller/src/dns/main.cc 98 2025-09-11 Thu 04:17:06:430.801 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.virtual_network_network_ipam.0, counting 0 elements controller/src/dns/main.cc 98 2025-09-11 Thu 04:17:06:430.810 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.global_qos_config.0, counting 1 elements controller/src/dns/main.cc 98 2025-09-11 Thu 04:17:06:430.818 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: VnniConfig has been updated after the bulk sync, the number of elements: 0 controller/src/dns/main.cc 109 2025-09-11 Thu 04:17:06:430.827 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: IpamConfig has been updated after the bulk sync, the number of elements: 2 controller/src/dns/main.cc 120 2025-09-11 Thu 04:17:06:430.835 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: VirtualDnsConfig has been updated after the bulk sync, the number of elements: 0 controller/src/dns/main.cc 131 2025-09-11 Thu 04:17:06:430.843 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: VirtualDnsRecordConfig has been updated after the bulk sync, the number of elements: 0 controller/src/dns/main.cc 142 2025-09-11 Thu 04:17:06:430.851 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: GlobalQosConfig has been updated after the bulk sync controller/src/dns/main.cc 152 2025-09-11 Thu 04:17:06:430.899 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection down src/contrail-common/config-client-mgr/config_amqp_client.cc 159 2025-09-11 Thu 04:17:06:430.910 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: Connect to Rabbit MQ with queue_delete TRUE src/contrail-common/config-client-mgr/config_amqp_client.cc 168 2025-09-11 Thu 04:17:06:435.887 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:06:435.941 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: SANDESH: Send FAILED: 1757564226430883 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Down description = RabbitMQ connection down ], ] ] flag_infos= [ [ ] ] description = Collector, Database:RabbitMQ connection down ], ] ] ] 2025-09-11 Thu 04:17:06:435.958 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:06:666.335 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection established src/contrail-common/config-client-mgr/config_amqp_client.cc 152 2025-09-11 Thu 04:17:06:666.427 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:06:666.438 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:06:666.470 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:06:666.499 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: SANDESH: Send FAILED: 1757564226666295 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:06:666.509 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:06:666.518 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:06:666.526 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:06:666.651 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:06:666.689 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Idle 2025-09-11 Thu 04:17:06:666.761 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:06:666.912 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: SANDESH: Send FAILED: 1757564226666607 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 305, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:06:666.940 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:06:666.950 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:06:666.972 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: SANDESH: Send FAILED: 1757564226666632 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:06:666.979 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:06:666.987 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:06:667.036 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:06:667.165 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: SANDESH: Send FAILED: 1757564226666740 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 306, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:06:667.180 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:06:667.194 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448725063424, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: TRUE src/contrail-common/config-client-mgr/config_client_manager.cc 219 2025-09-11 Thu 04:17:06:667.345 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: BIND named up; DNS is operational controller/src/dns/mgr/dns_mgr.cc 737 2025-09-11 Thu 04:17:07:667.549 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:17:07:667.816 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Connect : Start Connect timer 10.0.0.48:8086 2025-09-11 Thu 04:17:07:667.930 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:07:667.961 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: SANDESH: Send FAILED: 1757564227667619 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.48:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:07:667.971 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:07:667.981 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:07:668.125 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: SANDESH: Send FAILED: 1757564227667912 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 152 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 308, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 13, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:07:668.145 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:07:669.556 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:07:669.577 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:07:669.610 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:07:669.620 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:07:669.751 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:07:669.807 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Idle 2025-09-11 Thu 04:17:07:670.178 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:07:670.325 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: SANDESH: Send FAILED: 1757564227669701 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 152 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 308, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:07:670.343 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:07:670.353 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:07:670.375 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: SANDESH: Send FAILED: 1757564227669729 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:07:670.382 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:07:670.391 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:07:670.417 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:07:671.414 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: SANDESH: Send FAILED: 1757564227670162 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 152 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 309, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:07:671.447 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:07:837.676 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:07:955.400 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: DnsAgent [SYS_NOTICE]: DnsOperational: Bulk sync has been done (the DB has been synced with named) controller/src/dns/mgr/dns_mgr.cc 455 2025-09-11 Thu 04:17:09:669.913 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:17:09:670.188 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Connect : Start Connect timer 10.0.0.22:8086 2025-09-11 Thu 04:17:09:670.318 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:09:670.349 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: SANDESH: Send FAILED: 1757564229670013 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:09:670.367 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:09:670.382 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:09:670.525 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: SANDESH: Send FAILED: 1757564229670297 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 153 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 311, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:09:670.542 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:09:671.126 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:09:671.136 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:09:671.178 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:09:671.194 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:09:671.348 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:09:671.383 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Idle 2025-09-11 Thu 04:17:09:671.440 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:09:671.582 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: SANDESH: Send FAILED: 1757564229671312 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 153 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 311, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:09:671.597 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:09:671.607 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:09:671.628 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: SANDESH: Send FAILED: 1757564229671333 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:09:671.637 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:09:671.646 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:09:671.666 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:09:671.825 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: SANDESH: Send FAILED: 1757564229671429 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 153 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 312, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:09:671.840 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:10:872.498 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:10:872.679 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating access-control-list:default-domain:default-project:__no_rule__:ingress-access-control-list controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:10:872.728 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating security-group-access-control-list controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:10:927.429 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:10:929.006 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating access-control-list:default-domain:default-project:__no_rule__:egress-access-control-list controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:10:931.941 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating security-group-access-control-list controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:10:995.815 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:019.142 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:019.358 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 2e1fc63d-6d97-43e5-861a-2c3093d27091 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:11:057.238 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:081.848 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:081.996 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance a79654b7-d507-44f7-bbb7-2b6bf4c79ce3 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:11:128.903 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:165.915 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:166.023 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 85522317-b3f9-447d-bea5-72a8afc7db3b controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:11:223.972 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:256.553 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:256.656 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 010c7f62-733c-4f41-aaaa-9712d470b6e4 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:11:385.800 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:386.161 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating access-control-list:default-domain:default-project:ip-fabric:ip-fabric controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:11:386.225 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-network-access-control-list controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:11:421.992 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:443.873 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:022.583 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:084.287 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:115.321 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:130.706 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:164.133 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:178.234 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:216.754 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:216.881 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router ae44b01e-9ba6-46ee-b793-23f3d46999ec controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:219.293 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:219.378 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router d9886334-d660-45c3-97f9-3f7c9c1d2a94 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:326.296 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:326.389 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router 69d58baa-ec2b-4f35-bbd7-4ec0ee893627 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:327.400 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:327.483 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router d9886334-d660-45c3-97f9-3f7c9c1d2a94 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:399.376 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:399.523 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router 69d58baa-ec2b-4f35-bbd7-4ec0ee893627 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:413.963 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:414.072 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router ae44b01e-9ba6-46ee-b793-23f3d46999ec controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:445.175 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:475.267 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:517.198 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:671.472 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:17:13:671.698 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Connect : Start Connect timer 10.0.0.254:8086 2025-09-11 Thu 04:17:13:671.720 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:13:671.748 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:13:671.816 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:13:671.846 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: SANDESH: Send FAILED: 1757564233671538 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:13:671.855 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:13:671.865 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:13:671.872 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:13:671.973 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:13:672.002 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Idle 2025-09-11 Thu 04:17:13:672.069 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:13:672.228 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: SANDESH: Send FAILED: 1757564233671801 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 181 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 356, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:13:672.250 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:13:672.260 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:13:672.392 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: SANDESH: Send FAILED: 1757564233671933 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 181 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 356, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:13:672.406 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:13:672.415 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:13:672.435 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: SANDESH: Send FAILED: 1757564233671952 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:13:672.442 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:13:672.451 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:13:672.468 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:13:672.607 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: SANDESH: Send FAILED: 1757564233672060 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 181 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 357, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:13:672.622 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:14:485.602 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:14:546.608 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:14:558.549 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:15:361.861 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:21:672.086 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:17:21:672.354 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Connect : Start Connect timer 10.0.0.48:8086 2025-09-11 Thu 04:17:21:672.471 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:21:672.503 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: SANDESH: Send FAILED: 1757564241672176 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.48:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:21:672.513 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:21:672.522 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:21:672.688 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: SANDESH: Send FAILED: 1757564241672459 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 361, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:21:672.706 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:21:672.751 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:21:672.761 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:21:672.815 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:21:672.826 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:21:672.940 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:21:672.969 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: Idle 2025-09-11 Thu 04:17:21:673.031 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:21:673.180 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: SANDESH: Send FAILED: 1757564241672896 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 361, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:21:673.200 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:21:673.210 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:21:673.231 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: SANDESH: Send FAILED: 1757564241672919 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:21:673.238 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:21:673.246 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:21:673.268 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:21:673.404 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: SANDESH: Send FAILED: 1757564241673023 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 362, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:21:673.419 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:34:917.654 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:34:917.743 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: SANDESH: Send FAILED: 1757564254917517 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 33 dequeues = 33 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 49 dequeues = 49 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1757564241672968 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1757564241673266 ] max_sm_queue_count = 5 ] 2025-09-11 Thu 04:17:34:917.757 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:34:917.766 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:34:917.916 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: SANDESH: Send FAILED: 1757564254917632 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 362, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:34:917.941 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:37:673.064 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:17:37:673.286 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: Connect : Start Connect timer 10.0.0.22:8086 2025-09-11 Thu 04:17:37:673.409 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:37:673.439 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: SANDESH: Send FAILED: 1757564257673134 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:37:673.456 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:37:673.465 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:37:673.630 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: SANDESH: Send FAILED: 1757564257673394 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 362, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:37:673.645 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:37:674.599 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:37:674.609 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:37:674.628 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:37:674.637 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:37:674.747 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:37:674.774 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Idle 2025-09-11 Thu 04:17:37:674.846 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:37:674.990 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564257674708 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 362, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:37:675.007 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:37:675.018 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:37:675.040 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564257674732 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:37:675.047 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:37:675.055 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:37:675.077 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:37:675.246 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564257674837 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 363, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:37:675.266 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:42:501.236 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:42:501.387 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:42:501.425 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:42:517.913 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:42:518.051 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:42:518.089 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448609818368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:42:564.437 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:42:564.585 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:42:564.622 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:04:917.902 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:04:918.008 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: SANDESH: Send FAILED: 1757564284917732 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 40 dequeues = 40 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 59 dequeues = 59 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1757564257674773 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1757564257675076 ] max_sm_queue_count = 5 ] 2025-09-11 Thu 04:18:04:918.040 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:04:918.050 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:04:918.202 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: SANDESH: Send FAILED: 1757564284917868 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 188 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 372, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:04:918.218 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:05:509.667 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:05:509.724 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: SANDESH: Send FAILED: 1757564285509592 [SYS_NOTICE]: UveDnsInfo: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 start_time = 1757564225230371 build_info = {"build-info":[{"build-version":"v0+nightly.r24.1.centos","build-time":"2025-09-11 00:09:15.426696","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-10925-1.novalocal","build-id":"v0+nightly.r24.1.centos-nightly_r24.1_centos.el7@contrail","build-number":"unknown"}]} collector = 10.0.0.254:8086 self_ip_list= [ [ (*_iter12) = 10.20.0.17, ] ] ] 2025-09-11 Thu 04:18:05:509.740 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: UveDnsInfo 2025-09-11 Thu 04:18:07:674.927 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:18:07:675.276 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Connect : Start Connect timer 10.0.0.254:8086 2025-09-11 Thu 04:18:07:675.456 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:18:07:675.488 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: SANDESH: Send FAILED: 1757564287675067 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:18:07:675.499 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:18:07:675.509 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:18:07:675.707 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: SANDESH: Send FAILED: 1757564287675438 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 188 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 372, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:07:675.731 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:07:675.810 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:18:07:675.822 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:18:07:675.875 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:18:07:675.893 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:18:07:676.088 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:18:07:676.140 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Idle 2025-09-11 Thu 04:18:07:676.253 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:07:676.531 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564287676025 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 188 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 372, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:07:676.560 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:07:676.577 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:07:676.611 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564287676065 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:18:07:676.622 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:18:07:676.633 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:18:07:676.660 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:07:676.879 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564287676233 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 188 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 373, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:07:676.899 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:10:046.495 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:10:046.652 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:10:046.691 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:10:083.965 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:10:084.104 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:10:084.140 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:10:186.631 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:10:186.918 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:10:186.980 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:24:772.860 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:24:773.025 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-alarm-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:24:773.065 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-alarm-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:25:206.166 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:25:206.322 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-alarm-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:25:206.359 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-alarm-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:25:269.664 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:25:269.838 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-alarm-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:25:269.875 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448618215168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-alarm-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:34:581.852 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:34:582.029 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-snmp-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:34:582.076 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-snmp-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:34:630.347 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:34:630.513 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-snmp-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:34:630.569 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-snmp-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:34:659.566 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:34:659.741 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-snmp-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:34:659.800 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-snmp-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:34:918.121 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:34:918.200 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: SANDESH: Send FAILED: 1757564314917977 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 48 dequeues = 48 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 70 dequeues = 70 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1757564287676138 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1757564287676658 ] max_sm_queue_count = 5 ] 2025-09-11 Thu 04:18:34:918.214 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:34:918.223 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:34:918.378 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: SANDESH: Send FAILED: 1757564314918094 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 197 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 80 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 90 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 400, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:34:918.402 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:37:676.271 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:18:37:676.492 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Connect : Start Connect timer 10.0.0.48:8086 2025-09-11 Thu 04:18:37:676.609 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:18:37:676.640 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: SANDESH: Send FAILED: 1757564317676339 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.48:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:18:37:676.656 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:18:37:676.665 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:18:37:676.838 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: SANDESH: Send FAILED: 1757564317676595 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 197 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 80 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 90 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 400, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:37:676.856 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:37:678.967 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.254:47624::10.0.0.48:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-09-11 Thu 04:18:37:678.990 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: 10.0.0.254:47624::10.0.0.48:8086(-1) OnSessionEvent TCP Connected 2025-09-11 Thu 04:18:37:678.998 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: 10.0.0.254:47624::10.0.0.48:8086(-1):EvTcpConnected 2025-09-11 Thu 04:18:37:679.043 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvTcpConnected in state Connect 2025-09-11 Thu 04:18:37:679.055 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Connect : EvTcpConnected : Cancelling Connect timer 2025-09-11 Thu 04:18:37:679.087 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: ClientInit 2025-09-11 Thu 04:18:37:679.105 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-4080-1:contrail-dns:0:Control count 1 2025-09-11 Thu 04:18:37:679.206 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state ClientInit 2025-09-11 Thu 04:18:37:679.215 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-09-11 Thu 04:18:37:679.238 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564317679081 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.48:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:18:37:679.246 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Received UVE message in wrong state : NodeStatusUVE 2025-09-11 Thu 04:18:37:679.254 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state ClientInit 2025-09-11 Thu 04:18:37:679.261 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-09-11 Thu 04:18:37:679.271 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshSend in state ClientInit 2025-09-11 Thu 04:18:37:679.278 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-09-11 Thu 04:18:37:679.426 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SANDESH: Send FAILED: 1757564317679196 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = ClientInit successful_connections = 1 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 197 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 80 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 90 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 401, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:37:679.445 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-09-11 Thu 04:18:37:682.897 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: OnMessage control in state: ClientInit 2025-09-11 Thu 04:18:37:682.936 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-09-11 Thu 04:18:37:682.945 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: ClientInit : EvSandeshMessageRecv 2025-09-11 Thu 04:18:37:682.974 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Received Ctrl Message with size 0 2025-09-11 Thu 04:18:37:682.992 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SyncUVE Syncing ObjectDns val name = cn-jenkins-deploy-platform-ansible-os-4080-1 start_time = 1757564225230371 build_info = {"build-info":[{"build-version":"v0+nightly.r24.1.centos","build-time":"2025-09-11 00:09:15.426696","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-10925-1.novalocal","build-id":"v0+nightly.r24.1.centos-nightly_r24.1_centos.el7@contrail","build-number":"unknown"}]} collector = 10.0.0.254:8086 self_ip_list= [ [ (*_iter12) = 10.20.0.17, ] ] proxy seq 1 2025-09-11 Thu 04:18:37:683.004 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SyncAllMaps for DnsState:0 period 0 cycle 0 without seqno , total = 1 2025-09-11 Thu 04:18:37:683.214 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-4080-1:Control:contrail-dns:0 client_info= [ status = ClientInit successful_connections = 1 pid = 77295 http_port = 8092 start_time = 1757564224917242 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 48 dequeues = 48 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 70 dequeues = 70 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1757564287676138 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1757564287676658 ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 197 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 80 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 90 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 401, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 32 2025-09-11 Thu 04:18:37:683.240 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-09-11 Thu 04:18:37:683.264 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SyncUVE Syncing ObjectDns val name = cn-jenkins-deploy-platform-ansible-os-4080-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.48:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 22 2025-09-11 Thu 04:18:37:683.275 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-09-11 Thu 04:18:37:683.285 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SyncAllMaps for PeerFlapData:0 period 0 cycle 0 without seqno , total = 0 2025-09-11 Thu 04:18:37:683.292 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SyncAllMaps for PeerStatsData:0 period 0 cycle 0 without seqno , total = 0 2025-09-11 Thu 04:18:37:683.301 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: SyncAllMaps for XmppPeerInfoData:0 period 0 cycle 0 without seqno , total = 0 2025-09-11 Thu 04:18:37:683.329 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: Established 2025-09-11 Thu 04:20:11:133.124 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: XMPP [SYS_DEBUG]: XmppCreateConnection: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-1:10.20.0.25 SEND Xmpp creating dynamic channel 10.20.0.17:8093::10.20.0.25:44245 controller/src/xmpp/xmpp_server.cc 586 2025-09-11 Thu 04:20:11:133.193 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: XMPP [SYS_INFO]: XmppConnectionCreate: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-1:10.20.0.25 RECV Created Xmpp Server connection from cn-jenkins-deploy-platform-ansible-os-4080-1 To : controller/src/xmpp/xmpp_connection.cc 727 2025-09-11 Thu 04:20:11:133.252 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.17:8093::10.20.0.25:44245 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:20:11:133.345 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-1:10.20.0.25 RECV Mode Server: PassiveOpen in state: Idle peer ip: 10.20.0.25 ( ) controller/src/xmpp/xmpp_state_machine.cc 1326 2025-09-11 Thu 04:20:11:133.465 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:11:133.617 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = EvTcpPassiveOpen local_port = 8093 remote_port = 44245 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:11:159.006 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: XMPP [SYS_DEBUG]: XmppCreateConnection: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-1:10.20.0.193 SEND Xmpp creating dynamic channel 10.20.0.17:8093::10.20.0.193:37191 controller/src/xmpp/xmpp_server.cc 586 2025-09-11 Thu 04:20:11:159.070 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: XMPP [SYS_INFO]: XmppConnectionCreate: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-1:10.20.0.193 RECV Created Xmpp Server connection from cn-jenkins-deploy-platform-ansible-os-4080-1 To : controller/src/xmpp/xmpp_connection.cc 727 2025-09-11 Thu 04:20:11:159.123 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.17:8093::10.20.0.193:37191 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:20:11:159.181 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-1:10.20.0.193 RECV Mode Server: PassiveOpen in state: Idle peer ip: 10.20.0.193 ( ) controller/src/xmpp/xmpp_state_machine.cc 1326 2025-09-11 Thu 04:20:11:159.261 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:11:159.330 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = EvTcpPassiveOpen local_port = 8093 remote_port = 37191 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:11:359.806 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Established event = EvXmppOpen identifier = an-jenkins-deploy-platform-ansible-os-4080-1/dns ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:11:381.519 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448626611968, Pid 77295]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Established event = EvXmppOpen identifier = an-jenkins-deploy-platform-ansible-os-4080-2/dns ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:12:950.179 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448729261824, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:12:997.856 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448614016768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:104.417 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:104.573 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-router:default-global-system-config:an-jenkins-deploy-platform-ansible-os-4080-2 controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:20:13:104.610 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448651802368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-virtual-router controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:154.001 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448643405568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:154.169 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-router:default-global-system-config:an-jenkins-deploy-platform-ansible-os-4080-1 controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:20:13:154.229 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448639207168, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-virtual-router controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:482.689 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:482.890 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-machine-interface:default-global-system-config:an-jenkins-deploy-platform-ansible-os-4080-2:vhost0 controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:20:13:482.934 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-router-virtual-machine-interface controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:482.962 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448635008768, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-machine-interface-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:542.388 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448647603968, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:596.489 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:596.660 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-machine-interface:default-global-system-config:an-jenkins-deploy-platform-ansible-os-4080-1:vhost0 controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:20:13:596.705 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-router-virtual-machine-interface controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:596.730 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448630810368, Pid 77295]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-machine-interface-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:667.369 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:14:012.025 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448622413568, Pid 77295]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:21:35:530.955 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.17:8092::10.20.0.14:47428 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:21:36:044.565 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.17:8092::10.20.0.14:47442 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:21:36:159.744 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.17:8092::10.20.0.254:49056 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:21:36:236.122 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.17:8092::10.20.0.17:44798 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:21:36:347.941 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.17:8092::10.20.0.254:49060 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:21:36:864.067 UTC cn-jenkins-deploy-platform-ansible-os-4080-1 [Thread 140448885188736, Pid 77295]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.17:8092::10.20.0.17:36222 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254