09/25/2025 01:22:33.553 7f9b9b4d65c0 [contrail-control-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/25/2025 01:22:33.553 7f9b9b4d65c0 [contrail-control-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/25/2025 01:22:33.553 7f9b9b4d65c0 [contrail-control-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/25/2025 01:22:33.580 7f9b9862d4a0 [contrail-control-nodemgr] [ERROR]: Cannot write http_port 8101 to /tmp/contrail-control-nodemgr.0.http_port 09/25/2025 01:22:33.583 7f9b9862d4a0 [contrail-control-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8101 09/25/2025 01:22:33.588 7f9b9862d860 [contrail-control-nodemgr] [INFO]: Processing event[EvStart] in state[none] 09/25/2025 01:22:33.589 7f9b9862d860 [contrail-control-nodemgr] [DEBUG]: Add uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:22:33.590 7f9b9862d860 [contrail-control-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/25/2025 01:22:33.590 7f9b9862d860 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/25/2025 01:22:33.590 7f9b9862d860 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:33.590 7f9b9862d860 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/25/2025 01:22:33.590 7f9b9862d860 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:33.590 7f9b9862d860 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:33.590 7f9b9862d860 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/25/2025 01:22:33.590 7f9b9862d860 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:33.686 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'contrail-control', 'group': 'contrail-control', 'pid': 101556201770623014594171402183044144803591724815650899442581683995319694340154, 'start': '1758763350000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-dns', 'group': 'contrail-dns', 'pid': 99371437667630328507321948095246764270271907631726427534916604933771195536212, 'start': '1758763349000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-named', 'group': 'contrail-named', 'pid': 98705763429511718508236046066185115665684367001026173697030144076442884051200, 'start': '1758763350000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-control-nodemgr', 'group': 'contrail-control-nodemgr', 'pid': 63187138739008759495593844356060325547320428406084981902984508098987293740387, 'start': '1758763349000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 09/25/2025 01:22:33.840 7f9b9b4d65c0 [contrail-control-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1521-1', _node_type='Control', _instance_id='0', _timestamp=1758763353840772, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-control-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 09/25/2025 01:22:33.841 7f9b9b4d65c0 [contrail-control-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1521-1', _node_type='Control', _instance_id='0', _timestamp=1758763353841228, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='contrail-control', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758763350000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-dns', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758763349000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-named', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758763350000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-control-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758763349000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-09-25 00:15:10.514512", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11279-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed\n"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 09/25/2025 01:22:33.843 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:33.844 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 build_info = {"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-09-25 00:15:10.514512", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11279-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed "}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = package contrail-lib is not installed installed_package_version = package contrail-lib is not installed >> 09/25/2025 01:22:33.844 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:33.844 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:33.844 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/25/2025 01:22:33.844 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:33.844 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:33.844 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 deleted = False process_info = [ << process_name = contrail-control process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758763350000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-dns process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758763349000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-named process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758763350000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-control-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758763349000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-09-25 00:15:10.514512", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11279-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed "}]} >> 09/25/2025 01:22:33.844 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:34.557 7f9b98566f40 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:22:37.591 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:22:37.593 7f9b98596180 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:22:37.593 7f9b98596180 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:22:37.593 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:37.593 7f9b98596180 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:22:37.593 7f9b98596180 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:37.593 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:22:37.593 7f9b98596180 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:22:37.593 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:37.593 7f9b98596180 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:22:37.593 7f9b98596180 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:37.594 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:37.594 7f9b98596180 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:22:37.594 7f9b98596180 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:41.594 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:22:41.595 7f9b98596180 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:22:41.595 7f9b98596180 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:22:41.595 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:41.595 7f9b98596180 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:22:41.595 7f9b98596180 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:41.595 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:22:41.596 7f9b98596180 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:22:41.596 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:41.596 7f9b98596180 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:22:41.596 7f9b98596180 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:41.596 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:41.596 7f9b98596180 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:22:41.596 7f9b98596180 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:41.659 7f9b9b4d65c0 [contrail-control-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-k8s-1521-1', _node_type='Control', _instance_id='0', _timestamp=1758763361659464, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='contrail-control', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758763350000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-dns', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758763349000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-named', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758763350000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-control-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1758763349000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-09-25 00:15:10.514512", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11279-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed\n"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 09/25/2025 01:22:41.659 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11974824, partition_space_available_1k=69115660, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=568120, cpu_share=0.0, mem_res=4628, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=8068, cpu_share=0.0, mem_res=4196, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12180, cpu_share=0.0, mem_res=8304, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53228, cpu_share=0.0, mem_res=41576, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=3397056, free=228304, buffers=435712, cached=12502740, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.7, five_min_avg=1.62, fifteen_min_avg=0.79, cpu_share=0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:22:41.660 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:41.660 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 deleted = False process_info = [ << process_name = contrail-control process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758763350000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-dns process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758763349000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-named process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758763350000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-control-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1758763349000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "v0+nightly", "build-time" : "2025-09-25 00:15:10.514512", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-11279-1.novalocal", "build-id" : "package contrail", "build-number" : "lib is not installed "}]} >> 09/25/2025 01:22:41.660 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:41.660 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:41.660 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 11974824 partition_space_available_1k = 69115660 percentage_partition_space_used = 15 >> } all_core_file_list = [ ] process_mem_cpu_usage = { contrail-control : << mem_virt = 568120 cpu_share = 0.0 mem_res = 4628 >>contrail-dns : << mem_virt = 8068 cpu_share = 0.0 mem_res = 4196 >>contrail-named : << mem_virt = 12180 cpu_share = 0.0 mem_res = 8304 >>contrail-control-nodemgr : << mem_virt = 53228 cpu_share = 0.0 mem_res = 41576 >> } system_mem_usage = << total = 16184208 used = 3397056 free = 228304 buffers = 435712 cached = 12502740 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.7 five_min_avg = 1.62 fifteen_min_avg = 0.79 cpu_share = 0 node_type = control-node >> >> 09/25/2025 01:22:41.660 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:45.597 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:22:45.598 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:22:45.598 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:22:45.598 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:45.599 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:22:45.599 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:45.599 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:22:45.599 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:22:45.599 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:45.599 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:22:45.599 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:45.599 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:45.599 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:22:45.599 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:49.599 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:22:49.600 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:22:49.600 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:22:49.601 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:53.603 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:22:53.604 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:22:53.604 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:22:53.605 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:57.607 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:22:57.608 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:22:57.609 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:01.611 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:01.612 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:01.613 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:01.614 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:05.614 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:05.615 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:05.615 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:05.615 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:05.615 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:05.616 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:05.616 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:05.616 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:05.616 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:05.616 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:05.616 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:05.616 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:05.616 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:05.616 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:09.619 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:09.620 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:09.620 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:09.620 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:09.620 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:09.620 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:09.620 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:09.620 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:09.621 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:09.621 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:09.621 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:09.621 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:09.621 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:09.621 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:13.625 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:13.625 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:13.625 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:13.626 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:17.630 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:17.630 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:17.631 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:21.632 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:21.633 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:21.633 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:21.633 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:21.633 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:21.633 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:21.633 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:21.634 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:21.634 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:21.634 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:21.634 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:21.634 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:21.634 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:21.634 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:25.636 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:25.637 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:25.637 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:25.637 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:25.637 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:25.638 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:25.638 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:25.638 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:25.638 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:25.638 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:25.638 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:25.638 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:25.638 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:25.638 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:29.641 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:29.642 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:29.642 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:29.642 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:29.642 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:29.642 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:29.642 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:29.642 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:29.642 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:29.642 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:29.642 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:29.643 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:29.643 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:29.643 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:33.643 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:33.644 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:33.645 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:33.645 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:33.645 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:33.645 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:33.645 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:33.645 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:33.645 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:33.645 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:33.645 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:33.646 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:33.646 7f9b98596e00 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:33.646 7f9b98596e00 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:37.645 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:37.646 7f9b98596180 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:37.646 7f9b98596180 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:37.646 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:37.646 7f9b98596180 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:37.647 7f9b98596180 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:37.647 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:37.647 7f9b98596180 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:37.647 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:37.647 7f9b98596180 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:37.647 7f9b98596180 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:37.647 7f9b98596180 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:37.647 7f9b98596180 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:37.647 7f9b98596180 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:41.647 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12015524, partition_space_available_1k=69074960, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=530244, cpu_share=0.01, mem_res=8836, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16040, cpu_share=0.01, mem_res=10948, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13132, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53100, cpu_share=0.0, mem_res=41516, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=4494420, free=254396, buffers=426288, cached=11320316, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.99, five_min_avg=2.46, fifteen_min_avg=1.13, cpu_share=11.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:23:41.648 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:41.648 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 12015524 partition_space_available_1k = 69074960 percentage_partition_space_used = 15 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 530244 cpu_share = 0.01 mem_res = 8836 >>contrail-dns : << mem_virt = 16040 cpu_share = 0.01 mem_res = 10948 >>contrail-named : << mem_virt = 13132 cpu_share = 0.0 mem_res = 9096 >>contrail-control-nodemgr : << mem_virt = 53100 cpu_share = 0.0 mem_res = 41516 >> } system_mem_usage = << total = 16184208 used = 4494420 free = 254396 buffers = 426288 cached = 11320316 node_type = control-node >> system_cpu_usage = << one_min_avg = 4.99 five_min_avg = 2.46 fifteen_min_avg = 1.13 cpu_share = 11.25 node_type = control-node >> >> 09/25/2025 01:23:41.648 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:41.648 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:41.649 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:41.649 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:41.649 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:41.649 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:41.649 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:41.649 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:41.649 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:41.649 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:41.649 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:41.649 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:41.650 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:41.650 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:41.650 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:45.651 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:45.652 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:45.653 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:49.653 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:49.654 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:49.654 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:49.654 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:49.654 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:49.654 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:49.654 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:49.655 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:49.655 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:49.655 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:49.655 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:49.655 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:49.655 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:49.655 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:53.656 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:53.656 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:53.657 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:57.660 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:23:57.661 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:23:57.661 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:23:57.662 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:01.665 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:01.666 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:01.666 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:01.666 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:01.666 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:01.667 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:01.667 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:01.667 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:01.667 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:01.667 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:01.667 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:01.667 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:01.667 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:01.667 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:05.667 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:05.668 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:05.669 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:09.671 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:09.672 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:09.673 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:13.676 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:13.677 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:13.677 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:13.677 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:13.677 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:13.677 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:13.677 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:13.677 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:13.677 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:13.677 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:13.677 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:13.678 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:13.678 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:13.678 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:17.679 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:17.680 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:17.680 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:17.681 7f9b98566f40 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:21.684 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:21.685 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:21.685 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:21.685 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:21.685 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:21.685 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:21.685 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:21.685 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:21.685 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:21.685 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:21.686 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:21.686 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:21.686 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:21.686 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:25.688 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:25.689 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:25.690 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:25.690 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:25.690 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:25.690 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:25.690 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:25.690 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:25.691 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:25.691 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:25.691 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:25.691 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:25.691 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:25.691 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:29.694 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:29.695 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:29.695 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:29.695 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:29.695 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:29.695 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:29.695 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:29.696 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:29.696 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:29.696 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:29.696 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:29.696 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:29.696 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:29.696 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:33.699 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:33.700 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:33.700 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:33.700 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:33.700 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:33.700 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:33.700 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:33.700 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:33.701 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:33.701 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:33.701 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:33.701 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:33.701 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:33.701 7f9b98596ae0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:37.700 7f9b98596d60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:37.701 7f9b98596d60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:37.702 7f9b98596d60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:37.702 7f9b98596d60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:37.702 7f9b98596d60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:37.702 7f9b98596d60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:37.702 7f9b98596d60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:37.702 7f9b98596d60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:37.702 7f9b98596d60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:37.702 7f9b98596d60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:37.702 7f9b98596d60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:37.703 7f9b98596d60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:37.703 7f9b98596d60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:37.703 7f9b98596d60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:41.649 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12018748, partition_space_available_1k=69071736, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=530580, cpu_share=0.0, mem_res=9220, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16408, cpu_share=0.0, mem_res=11212, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53172, cpu_share=0.0, mem_res=41580, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=4538128, free=183228, buffers=427984, cached=11346160, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.47, five_min_avg=2.51, fifteen_min_avg=1.24, cpu_share=3.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:24:41.650 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:41.650 7f9b98596720 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 12018748 partition_space_available_1k = 69071736 percentage_partition_space_used = 15 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 530580 cpu_share = 0.0 mem_res = 9220 >>contrail-dns : << mem_virt = 16408 cpu_share = 0.0 mem_res = 11212 >>contrail-named : << mem_virt = 13164 cpu_share = 0.0 mem_res = 9096 >>contrail-control-nodemgr : << mem_virt = 53172 cpu_share = 0.0 mem_res = 41580 >> } system_mem_usage = << total = 16184208 used = 4538128 free = 183228 buffers = 427984 cached = 11346160 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.47 five_min_avg = 2.51 fifteen_min_avg = 1.24 cpu_share = 3.25 node_type = control-node >> >> 09/25/2025 01:24:41.650 7f9b98596720 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:41.702 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:41.703 7f9b98596720 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 09/25/2025 01:24:41.704 7f9b98596720 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:41.704 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:41.704 7f9b98596720 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:41.704 7f9b98596720 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:41.704 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/25/2025 01:24:41.704 7f9b98596720 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/25/2025 01:24:41.704 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:41.704 7f9b98596720 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/25/2025 01:24:41.704 7f9b98596720 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:41.704 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:41.705 7f9b98596720 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:41.705 7f9b98596720 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:43.398 7f9b98596720 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:24:45.706 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:45.707 7f9b98596720 [contrail-control-nodemgr] [INFO]: Session Event: TCP Connected 09/25/2025 01:24:45.707 7f9b98596720 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:45.708 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:45.708 7f9b98596720 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:45.708 7f9b98596720 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:45.708 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/25/2025 01:24:45.708 7f9b98596720 [contrail-control-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 09/25/2025 01:24:45.708 7f9b98596720 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/25/2025 01:24:45.708 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/25/2025 01:24:45.708 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/25/2025 01:24:47.605 7f9b98596ae0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connection Closed 09/25/2025 01:24:47.605 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 09/25/2025 01:24:47.605 7f9b98596720 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 09/25/2025 01:24:47.605 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:47.605 7f9b98596720 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 09/25/2025 01:24:47.605 7f9b98596720 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:47.606 7f9b98596720 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:47.606 7f9b98596720 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1758763353590235 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/25/2025 01:24:47.606 7f9b98596720 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/25/2025 01:24:51.607 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/25/2025 01:24:51.608 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Session Event: TCP Connected 09/25/2025 01:24:51.609 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/25/2025 01:24:51.609 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:51.609 7f9b992cc5e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1521-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/25/2025 01:24:51.609 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/25/2025 01:24:51.609 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/25/2025 01:24:51.609 7f9b992cc5e0 [contrail-control-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 09/25/2025 01:24:51.609 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/25/2025 01:24:51.609 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/25/2025 01:24:51.609 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/25/2025 01:24:51.615 7f9b98596e00 [contrail-control-nodemgr] [DEBUG]: Extracted sandesh length: 0000001013 09/25/2025 01:24:51.616 7f9b98596e00 [contrail-control-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 09/25/2025 01:24:51.616 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 09/25/2025 01:24:51.616 7f9b992cc5e0 [contrail-control-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 09/25/2025 01:24:51.616 7f9b992cc5e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:24:51.616 7f9b992cc5e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/25/2025 01:24:51.616 7f9b992cc5e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 09/25/2025 01:24:51.617 7f9b9862dcc0 [contrail-control-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/25/2025 01:24:51.617 7f9b9862dcc0 [contrail-control-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/25/2025 01:24:58.813 7f9b98596e00 [contrail-control-nodemgr] [DEBUG]: Extracted sandesh length: 0000001293 09/25/2025 01:24:58.813 7f9b98596e00 [contrail-control-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 09/25/2025 01:24:58.820 7f9b98566ea0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 09/25/2025 01:24:58.823 7f9b98566ea0 [contrail-control-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 2 09/25/2025 01:24:58.823 7f9b98566ea0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:24:58.823 7f9b98566ea0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/25/2025 01:25:01.858 7f9b98566ea0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:25:12.640 7f9b98566540 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:25:41.672 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12887852, partition_space_available_1k=68202632, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=324912, cpu_share=0.0, mem_res=9604, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17004, cpu_share=0.0, mem_res=11740, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52072, cpu_share=0.0, mem_res=41488, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7509580, free=248924, buffers=392808, cached=8371280, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.13, five_min_avg=3.21, fifteen_min_avg=1.56, cpu_share=16.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:25:45.661 7f9b985662c0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:25:48.403 7f9b985662c0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:25:54.989 7f9b985662c0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:26:33.771 7f9b98596720 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:26:37.628 7f9b98566720 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:26:40.326 7f9b98566720 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:26:41.681 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12921900, partition_space_available_1k=68168584, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=307436, cpu_share=0.02, mem_res=11316, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18240, cpu_share=0.02, mem_res=12796, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52164, cpu_share=0.0, mem_res=41424, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7861932, free=299572, buffers=382540, cached=8012500, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.65, five_min_avg=3.42, fifteen_min_avg=1.74, cpu_share=14.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:26:44.021 7f9b98566540 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:26:46.811 7f9b98566540 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:27:41.703 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12928220, partition_space_available_1k=68162264, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=307632, cpu_share=0.0, mem_res=11580, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18392, cpu_share=0.0, mem_res=12796, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52212, cpu_share=0.0, mem_res=41524, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7753032, free=321920, buffers=384372, cached=8031348, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.9, five_min_avg=3.18, fifteen_min_avg=1.76, cpu_share=8.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:28:41.708 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12935908, partition_space_available_1k=68154576, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=308708, cpu_share=0.0, mem_res=11976, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18452, cpu_share=0.01, mem_res=12928, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52284, cpu_share=0.0, mem_res=41424, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7841372, free=198784, buffers=388036, cached=8062432, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.3, five_min_avg=2.92, fifteen_min_avg=1.76, cpu_share=9.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:29:11.957 7f9b985667c0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:29:14.784 7f9b985667c0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:29:18.927 7f9b985667c0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:29:41.706 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12947736, partition_space_available_1k=68142748, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=310136, cpu_share=0.01, mem_res=12768, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18656, cpu_share=0.0, mem_res=13192, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52156, cpu_share=0.0, mem_res=41472, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7769008, free=256000, buffers=388176, cached=8077448, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.02, five_min_avg=2.73, fifteen_min_avg=1.76, cpu_share=9.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:30:41.577 7f9b98596720 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:30:41.719 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12954972, partition_space_available_1k=68135512, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=310468, cpu_share=0.0, mem_res=13156, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18816, cpu_share=0.0, mem_res=13324, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52220, cpu_share=0.0, mem_res=41580, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7831360, free=219392, buffers=386972, cached=8052844, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.52, five_min_avg=2.44, fifteen_min_avg=1.72, cpu_share=7.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:31:41.728 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12963672, partition_space_available_1k=68126812, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=310056, cpu_share=0.01, mem_res=13672, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18980, cpu_share=0.01, mem_res=13456, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52196, cpu_share=0.0, mem_res=41384, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7774452, free=268424, buffers=387464, cached=8059336, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.41, five_min_avg=2.23, fifteen_min_avg=1.69, cpu_share=7.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:32:41.778 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=13074356, partition_space_available_1k=68016128, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=307128, cpu_share=0.0, mem_res=14068, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19092, cpu_share=0.0, mem_res=13588, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52364, cpu_share=0.0, mem_res=41424, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7849544, free=243980, buffers=387088, cached=8009336, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.31, five_min_avg=2.27, fifteen_min_avg=1.74, cpu_share=9.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:33:41.761 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12980556, partition_space_available_1k=68109928, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=307416, cpu_share=0.0, mem_res=14200, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19188, cpu_share=0.0, mem_res=13720, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52392, cpu_share=0.0, mem_res=41512, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7793384, free=385528, buffers=387368, cached=7923800, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.58, five_min_avg=2.07, fifteen_min_avg=1.7, cpu_share=7.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:34:41.763 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12995708, partition_space_available_1k=68094776, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=307624, cpu_share=0.0, mem_res=14460, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19232, cpu_share=0.0, mem_res=13720, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52576, cpu_share=0.0, mem_res=41604, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7801964, free=367420, buffers=387496, cached=7932448, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.2, five_min_avg=1.87, fifteen_min_avg=1.66, cpu_share=7.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:35:06.177 7f9b98566cc0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:35:09.310 7f9b9862dea0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:35:14.407 7f9b9862dea0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:35:27.975 7f9b98566040 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:35:41.778 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14216460, partition_space_available_1k=66874024, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=251804, cpu_share=0.0, mem_res=14592, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19320, cpu_share=0.0, mem_res=13720, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52188, cpu_share=0.0, mem_res=41536, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=8064088, free=281840, buffers=395904, cached=7760800, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.07, five_min_avg=2.27, fifteen_min_avg=1.81, cpu_share=12.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:36:41.781 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14268464, partition_space_available_1k=66822020, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=253640, cpu_share=0.02, mem_res=15644, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19844, cpu_share=0.02, mem_res=14248, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52184, cpu_share=0.0, mem_res=41476, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=8032520, free=249836, buffers=404040, cached=7815880, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.42, five_min_avg=2.62, fifteen_min_avg=1.96, cpu_share=10.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:37:41.804 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14282860, partition_space_available_1k=66807624, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=253296, cpu_share=0.02, mem_res=16940, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21020, cpu_share=0.02, mem_res=14908, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52204, cpu_share=0.0, mem_res=41536, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=8046632, free=244988, buffers=409328, cached=7801820, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.33, five_min_avg=3.78, fifteen_min_avg=2.41, cpu_share=12.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:38:41.789 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14288472, partition_space_available_1k=66802012, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=254092, cpu_share=0.01, mem_res=17712, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21172, cpu_share=0.0, mem_res=15172, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52156, cpu_share=0.0, mem_res=41512, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=8059564, free=221856, buffers=409584, cached=7812024, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.01, five_min_avg=3.61, fifteen_min_avg=2.44, cpu_share=8.0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:39:41.800 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14291036, partition_space_available_1k=66799448, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=252532, cpu_share=0.0, mem_res=17844, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21360, cpu_share=0.01, mem_res=15304, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52180, cpu_share=0.0, mem_res=41552, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=8069764, free=254024, buffers=407756, cached=7771760, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.24, five_min_avg=3.18, fifteen_min_avg=2.37, cpu_share=8.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:40:03.808 7f9b9862dea0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:40:06.626 7f9b9862dea0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:40:11.224 7f9b992cc040 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:40:41.814 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14298536, partition_space_available_1k=66791948, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=250600, cpu_share=0.0, mem_res=18240, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21512, cpu_share=0.0, mem_res=15436, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52324, cpu_share=0.0, mem_res=41688, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=8069684, free=303948, buffers=404856, cached=7724604, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.26, five_min_avg=3.44, fifteen_min_avg=2.5, cpu_share=9.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:41:41.856 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14311636, partition_space_available_1k=66778848, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=252044, cpu_share=0.01, mem_res=18636, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22024, cpu_share=0.01, mem_res=15832, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52232, cpu_share=0.0, mem_res=41444, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=8043848, free=287868, buffers=408348, cached=7761836, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.03, five_min_avg=3.22, fifteen_min_avg=2.48, cpu_share=12.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:41:41.946 7f9b9862d860 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:42:41.840 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14914452, partition_space_available_1k=66176032, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=233784, cpu_share=0.0, mem_res=18768, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22020, cpu_share=0.0, mem_res=15832, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52144, cpu_share=0.0, mem_res=41320, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=8042816, free=270416, buffers=394488, cached=7799364, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.86, five_min_avg=3.11, fifteen_min_avg=2.49, cpu_share=9.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:42:55.617 7f9b9862dea0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:43:21.730 7f9b992cc5e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:43:41.844 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14923100, partition_space_available_1k=66167384, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=227956, cpu_share=0.01, mem_res=18880, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22040, cpu_share=0.01, mem_res=15832, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52164, cpu_share=0.0, mem_res=41408, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=8051816, free=361592, buffers=401276, cached=7694080, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.99, five_min_avg=2.8, fifteen_min_avg=2.42, cpu_share=9.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:43:48.631 7f9b992cc5e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:44:14.492 7f9b98566720 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:44:41.841 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14929888, partition_space_available_1k=66160596, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=226932, cpu_share=0.01, mem_res=19144, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22176, cpu_share=0.01, mem_res=15964, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52212, cpu_share=0.0, mem_res=41360, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=8016980, free=419792, buffers=399180, cached=7673248, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.35, five_min_avg=2.8, fifteen_min_avg=2.44, cpu_share=10.0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:45:41.850 7f9b9b4d65c0 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-k8s-1521-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14932980, partition_space_available_1k=66157504, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=227228, cpu_share=0.0, mem_res=19408, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22272, cpu_share=0.01, mem_res=15964, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13164, cpu_share=0.0, mem_res=9096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52228, cpu_share=0.0, mem_res=41344, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16184208, used=7920828, free=520928, buffers=399208, cached=7668008, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.66, five_min_avg=2.51, fifteen_min_avg=2.37, cpu_share=7.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 09/25/2025 01:45:53.415 7f9b9862dc20 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:45:56.825 7f9b98859360 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 09/25/2025 01:46:00.081 7f9b98859360 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map