03/31/2025 02:14:15.528 7f47cfd5cd48 [contrail-control-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/31/2025 02:14:15.528 7f47cfd5cd48 [contrail-control-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/31/2025 02:14:15.528 7f47cfd5cd48 [contrail-control-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/31/2025 02:14:15.551 7f47cc3ee048 [contrail-control-nodemgr] [ERROR]: Cannot write http_port 8101 to /tmp/contrail-control-nodemgr.0.http_port 03/31/2025 02:14:15.558 7f47cc3ee048 [contrail-control-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8101 03/31/2025 02:14:15.568 7f47cc3e1978 [contrail-control-nodemgr] [INFO]: Processing event[EvStart] in state[none] 03/31/2025 02:14:15.571 7f47cc3e1978 [contrail-control-nodemgr] [DEBUG]: Add uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:14:15.580 7f47cc3e1978 [contrail-control-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/31/2025 02:14:15.580 7f47cc3e1978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/31/2025 02:14:15.580 7f47cc3e1978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.581 7f47cc3e1978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/31/2025 02:14:15.581 7f47cc3e1978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.582 7f47cc3e1978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.583 7f47cc3e1978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/31/2025 02:14:15.583 7f47cc3e1978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.765 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'contrail-control', 'group': 'contrail-control', 'pid': 38108439760816989576552346444789768894413604680429795517296919916723431152942, 'start': '1743387252000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-dns', 'group': 'contrail-dns', 'pid': 57437354597843261383580698069160559517807821552160974960591176224233822165704, 'start': '1743387252000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-named', 'group': 'contrail-named', 'pid': 17015805120449799502250776139310149297875046179422251615690293610918915691047, 'start': '1743387253000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-control-nodemgr', 'group': 'contrail-control-nodemgr', 'pid': 78464753796423846523564441929749468277833073310286972489926615129852251846183, 'start': '1743387252000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 03/31/2025 02:14:15.926 7f47cfd5cd48 [contrail-control-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='Control', _instance_id='0', _timestamp=1743387255926193, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-control-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 03/31/2025 02:14:15.926 7f47cfd5cd48 [contrail-control-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='Control', _instance_id='0', _timestamp=1743387255926756, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='contrail-control', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387252000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-dns', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387252000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-named', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387253000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-control-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387252000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-31 00:53:57.142496", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8423-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 03/31/2025 02:14:15.931 7f47d5ff1a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.932 7f47d5ff1a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-31 00:53:57.142496", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8423-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 03/31/2025 02:14:15.932 7f47d5ff1a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.932 7f47d5ff1a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.932 7f47d5ff1a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/31/2025 02:14:15.933 7f47d5ff1a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.933 7f47d5ff1a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:15.933 7f47d5ff1a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 deleted = False process_info = [ << process_name = contrail-control process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387252000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-dns process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387252000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-named process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387253000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-control-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387252000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-31 00:53:57.142496", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8423-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 03/31/2025 02:14:15.933 7f47d5ff1a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:19.007 7f47cc2506d8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:14:19.571 7f47cd76b6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:19.573 7f47cd76b6d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:19.573 7f47cd76b6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:19.573 7f47cd76b6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:19.573 7f47cd76b6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:19.573 7f47cd76b6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:19.573 7f47cd76b6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:19.574 7f47cd76b6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:19.574 7f47cd76b6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:19.574 7f47cd76b6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:19.574 7f47cd76b6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:19.574 7f47cd76b6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:19.574 7f47cd76b6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 03/31/2025 02:14:19.574 7f47cd76b6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:23.576 7f47cc250438 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:23.577 7f47cc250438 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:23.577 7f47cc250438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:23.577 7f47cc250438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:23.577 7f47cc250438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:23.577 7f47cc250438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:23.577 7f47cc250438 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:23.578 7f47cc250438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:23.578 7f47cc250438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:23.578 7f47cc250438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:23.578 7f47cc250438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:23.578 7f47cc250438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:23.578 7f47cc250438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 03/31/2025 02:14:23.578 7f47cc250438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:23.710 7f47cfd5cd48 [contrail-control-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='Control', _instance_id='0', _timestamp=1743387263710195, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='contrail-control', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387252000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-dns', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387252000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-named', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387253000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-control-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743387252000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-31 00:53:57.142496", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8423-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 03/31/2025 02:14:23.710 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50232024, partition_space_available_1k=30858460, percentage_partition_space_used=62, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=6296, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=8748, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33384, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51108, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=11373396, free=579044, buffers=766704, cached=19945968, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.15, five_min_avg=3.29, fifteen_min_avg=2.26, cpu_share=0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:14:23.710 7f47cd790b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:23.711 7f47cd790b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 deleted = False process_info = [ << process_name = contrail-control process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387252000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-dns process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387252000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-named process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387253000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-control-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743387252000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-03-31 00:53:57.142496", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8423-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 03/31/2025 02:14:23.711 7f47cd790b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:23.711 7f47cd790b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:23.711 7f47cd790b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50232024 partition_space_available_1k = 30858460 percentage_partition_space_used = 62 >> } all_core_file_list = [ ] process_mem_cpu_usage = { contrail-control : << mem_virt = 6296 cpu_share = 0 mem_res = 0 >>contrail-dns : << mem_virt = 8748 cpu_share = 0 mem_res = 0 >>contrail-named : << mem_virt = 33384 cpu_share = 0 mem_res = 0 >>contrail-control-nodemgr : << mem_virt = 51108 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 11373396 free = 579044 buffers = 766704 cached = 19945968 node_type = control-node >> system_cpu_usage = << one_min_avg = 5.15 five_min_avg = 3.29 fifteen_min_avg = 2.26 cpu_share = 0 node_type = control-node >> >> 03/31/2025 02:14:23.711 7f47cd790b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:25.525 7f47cf7ccd68 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:14:27.578 7f47cf7ccd68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:27.579 7f47cf7ccd68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:27.579 7f47cf7ccd68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:27.579 7f47cf7ccd68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:27.579 7f47cf7ccd68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:27.580 7f47cf7ccd68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:27.580 7f47cf7ccd68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:27.580 7f47cf7ccd68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:27.580 7f47cf7ccd68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:27.580 7f47cf7ccd68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:27.580 7f47cf7ccd68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:27.580 7f47cf7ccd68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:27.580 7f47cf7ccd68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 03/31/2025 02:14:27.580 7f47cf7ccd68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:31.322 7f47d1259828 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:14:31.580 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:31.581 7f47d1259828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:31.581 7f47d1259828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:31.581 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:31.581 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:31.581 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:31.581 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:31.582 7f47d1259828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:31.582 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:31.582 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:31.582 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:31.582 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:31.582 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 03/31/2025 02:14:31.582 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:35.585 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:35.586 7f47d1259828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:35.587 7f47d1259828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:35.587 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:35.587 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:35.587 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:35.587 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:35.587 7f47d1259828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:35.587 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:35.588 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:35.588 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:35.588 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:35.588 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 03/31/2025 02:14:35.588 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:39.589 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:39.590 7f47d1259828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:39.590 7f47d1259828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:39.590 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:39.590 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:39.590 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:39.590 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:39.591 7f47d1259828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:39.591 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:39.591 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:39.591 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:39.591 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:39.591 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 03/31/2025 02:14:39.591 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:42.547 7f47d1259828 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:14:43.591 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:43.593 7f47d1259828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:14:43.593 7f47d1259828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:43.593 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:43.593 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:43.593 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:43.593 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:14:43.594 7f47d1259828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:14:43.594 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:43.594 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:14:43.594 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:43.594 7f47d1259828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:43.594 7f47d1259828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 03/31/2025 02:14:43.594 7f47d1259828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:47.594 7f47cd78b828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:48.436 7f47cd78b828 [contrail-control-nodemgr] [INFO]: Session Event: TCP Connected 03/31/2025 02:14:48.437 7f47cd78b828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:48.437 7f47cd78b828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:48.437 7f47cd78b828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:14:48.437 7f47cd78b828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:48.437 7f47cd78b828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/31/2025 02:14:48.437 7f47cd78b828 [contrail-control-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/31/2025 02:14:48.439 7f47cd78b828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/31/2025 02:14:48.439 7f47cd78b828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/31/2025 02:14:48.440 7f47cd78b828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/31/2025 02:14:48.447 7f47cd790a20 [contrail-control-nodemgr] [DEBUG]: Extracted sandesh length: 0000001011 03/31/2025 02:14:48.447 7f47cd790a20 [contrail-control-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/31/2025 02:14:48.448 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/31/2025 02:14:48.448 7f47d1208c18 [contrail-control-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/31/2025 02:14:48.448 7f47d1208c18 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:14:48.448 7f47d1208c18 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/31/2025 02:14:48.448 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/31/2025 02:14:48.449 7f47cc250438 [contrail-control-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/31/2025 02:14:48.450 7f47cc250438 [contrail-control-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/31/2025 02:15:07.584 7f47cfb8e780 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:15:23.645 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50833944, partition_space_available_1k=30256540, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11976, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14804, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=38484, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42072, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14583100, free=11047068, buffers=803688, cached=6231256, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.84, five_min_avg=4.12, fifteen_min_avg=2.61, cpu_share=8.12, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:15:26.377 7f47cf7cc438 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:15:41.263 7f47cf8b34e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:16:02.640 7f47cfb7aac8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:16:13.216 7f47cfb7aac8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:16:23.651 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50848528, partition_space_available_1k=30241956, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13472, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15580, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39328, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42080, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14648008, free=9727668, buffers=841028, cached=7448408, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.52, five_min_avg=5.33, fifteen_min_avg=3.13, cpu_share=5.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:16:54.821 7f47cfd56198 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:17:23.658 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50855496, partition_space_available_1k=30234988, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13748, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15836, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39328, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42088, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14625908, free=9708664, buffers=842904, cached=7487636, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.53, five_min_avg=4.77, fifteen_min_avg=3.07, cpu_share=2.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:18:23.679 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50858856, partition_space_available_1k=30231628, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14172, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16072, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39328, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42080, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14616480, free=9712460, buffers=842976, cached=7493196, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.75, five_min_avg=4.23, fifteen_min_avg=2.99, cpu_share=2.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:19:04.570 7f47cc250e10 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:19:13.328 7f47cc250e10 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:19:15.988 7f47cfd5cd48 [contrail-control-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='Control', _instance_id='0', _timestamp=1743387555988252, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-control-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.17:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='NTP state unsynchronized. ')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 03/31/2025 02:19:23.705 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50866788, partition_space_available_1k=30223696, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14440, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16360, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39328, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42100, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14646044, free=9672556, buffers=843116, cached=7503396, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.16, five_min_avg=3.76, fifteen_min_avg=2.91, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:19:24.925 7f47cfb8eb70 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:19:47.807 7f47cd77d0f0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:20:23.683 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50880716, partition_space_available_1k=30209768, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15068, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16548, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42100, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14735600, free=9479956, buffers=843172, cached=7606384, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.52, five_min_avg=4.59, fifteen_min_avg=3.26, cpu_share=5.88, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:21:23.689 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50878256, partition_space_available_1k=30212228, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15356, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16724, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42100, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14647848, free=9495200, buffers=843180, cached=7678884, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.21, five_min_avg=4.41, fifteen_min_avg=3.28, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:22:23.699 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50882324, partition_space_available_1k=30208160, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15664, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16964, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42104, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14650136, free=9487840, buffers=843180, cached=7683956, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.67, five_min_avg=3.96, fifteen_min_avg=3.2, cpu_share=2.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:23:23.709 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50885864, partition_space_available_1k=30204620, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15844, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17100, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42104, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14643648, free=9490036, buffers=843192, cached=7688236, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.9, five_min_avg=3.52, fifteen_min_avg=3.09, cpu_share=2.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:23:40.574 7f47cf7ccd68 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:23:49.169 7f47cf8b34e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:24:04.680 7f47cd6b30f0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:24:14.257 7f47d1259f60 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:24:16.025 7f47cfd5cd48 [contrail-control-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-494-1', _node_type='Control', _instance_id='0', _timestamp=1743387856025387, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-control-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.17:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 03/31/2025 02:24:23.728 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50898856, partition_space_available_1k=30191628, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16132, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17256, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42104, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14709264, free=9406840, buffers=843244, cached=7705764, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.73, five_min_avg=4.56, fifteen_min_avg=3.47, cpu_share=4.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:25:23.726 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50899828, partition_space_available_1k=30190656, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16288, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17456, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42108, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14692844, free=9416484, buffers=843244, cached=7712540, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.42, five_min_avg=4.0, fifteen_min_avg=3.35, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:26:23.732 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51006236, partition_space_available_1k=30084248, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16460, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17628, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42112, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14679528, free=9423112, buffers=843248, cached=7719224, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.16, five_min_avg=3.83, fifteen_min_avg=3.33, cpu_share=2.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:27:23.743 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50907520, partition_space_available_1k=30182964, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16684, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17700, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42112, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14670484, free=9495096, buffers=843248, cached=7656284, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.4, five_min_avg=3.49, fifteen_min_avg=3.25, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:28:02.965 7f47d0df5438 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:28:11.469 7f47cd7fd780 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:28:22.569 7f47cd6b0cc0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:28:23.736 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50915548, partition_space_available_1k=30174936, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16936, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17876, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42116, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14745948, free=9411044, buffers=843248, cached=7664872, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.77, five_min_avg=3.38, fifteen_min_avg=3.22, cpu_share=2.88, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:28:32.134 7f47d1259828 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:29:23.750 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50914920, partition_space_available_1k=30175564, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17252, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18020, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42120, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14683768, free=9472880, buffers=843248, cached=7665216, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.01, five_min_avg=3.02, fifteen_min_avg=3.1, cpu_share=2.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:30:23.749 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50924408, partition_space_available_1k=30166076, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17504, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18096, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42116, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14714596, free=9422360, buffers=843252, cached=7684904, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.42, five_min_avg=3.25, fifteen_min_avg=3.18, cpu_share=4.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:31:23.755 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50929124, partition_space_available_1k=30161360, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17740, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18212, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42124, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14698096, free=9434272, buffers=843280, cached=7689464, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.39, five_min_avg=2.98, fifteen_min_avg=3.08, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:32:23.754 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50932712, partition_space_available_1k=30157772, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17796, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18284, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42120, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14691888, free=9435104, buffers=843296, cached=7694824, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.33, five_min_avg=2.82, fifteen_min_avg=3.02, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:33:21.330 7f47cd71beb8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:33:23.758 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50936132, partition_space_available_1k=30154352, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17964, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18356, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42124, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14758340, free=9359404, buffers=843312, cached=7704056, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.9, five_min_avg=2.62, fifteen_min_avg=2.94, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:33:35.358 7f47cd7698d0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:33:43.945 7f47cd77d240 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:33:54.969 7f47d6b9a0f0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:34:23.772 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50940288, partition_space_available_1k=30150196, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18160, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18432, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42124, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14690728, free=9418376, buffers=843560, cached=7712448, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.6, five_min_avg=2.65, fifteen_min_avg=2.93, cpu_share=2.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:35:23.762 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50950376, partition_space_available_1k=30140108, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18344, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18492, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42128, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14721512, free=9349596, buffers=844576, cached=7749428, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.1, five_min_avg=3.21, fifteen_min_avg=3.11, cpu_share=4.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:36:23.765 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50956044, partition_space_available_1k=30134440, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18564, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18588, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42128, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14728088, free=9335328, buffers=844580, cached=7757116, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.75, five_min_avg=3.3, fifteen_min_avg=3.15, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:37:23.779 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50959536, partition_space_available_1k=30130948, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18888, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18668, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42128, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14740452, free=9317984, buffers=844584, cached=7762092, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.1, five_min_avg=2.88, fifteen_min_avg=3.01, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:37:44.014 7f47cd6c6a20 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:38:23.792 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50963124, partition_space_available_1k=30127360, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18804, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18748, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42136, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14723180, free=9330684, buffers=844592, cached=7766656, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.95, five_min_avg=2.68, fifteen_min_avg=2.93, cpu_share=2.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:39:08.192 7f47cd77de10 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:39:16.841 7f47cc3f54e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:39:23.804 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50973280, partition_space_available_1k=30117204, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19112, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18824, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42132, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14798436, free=9238312, buffers=844596, cached=7783768, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.71, five_min_avg=3.43, fifteen_min_avg=3.18, cpu_share=4.0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:39:28.136 7f47cd78a588 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:40:23.806 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50971892, partition_space_available_1k=30118592, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19220, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18904, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42136, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14843520, free=9189680, buffers=844596, cached=7787316, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.95, five_min_avg=3.14, fifteen_min_avg=3.09, cpu_share=2.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:41:23.817 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50976616, partition_space_available_1k=30113868, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19400, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18964, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42140, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14811176, free=9215028, buffers=844596, cached=7794312, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.42, five_min_avg=2.95, fifteen_min_avg=3.03, cpu_share=2.0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:42:05.306 7f47d5ff1a20 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:42:23.826 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50980516, partition_space_available_1k=30109968, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19656, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19028, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42140, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14813120, free=9207256, buffers=844596, cached=7800140, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.73, five_min_avg=2.93, fifteen_min_avg=3.02, cpu_share=2.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:43:23.836 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50984176, partition_space_available_1k=30106308, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19640, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19096, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42140, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14798156, free=9217672, buffers=844596, cached=7804688, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.32, five_min_avg=2.78, fifteen_min_avg=2.96, cpu_share=1.88, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:44:15.157 7f47cc3d9e10 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:44:23.802 7f47cd76b630 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:44:23.855 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50992288, partition_space_available_1k=30098196, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19944, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19168, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42140, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14921124, free=9085960, buffers=844596, cached=7813432, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.05, five_min_avg=2.85, fifteen_min_avg=2.97, cpu_share=2.12, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:44:38.002 7f47cd71e6d8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:45:23.858 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50993384, partition_space_available_1k=30097100, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19948, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19240, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42144, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14897404, free=9099964, buffers=844596, cached=7823148, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.55, five_min_avg=3.48, fifteen_min_avg=3.19, cpu_share=4.12, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:46:23.862 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50998188, partition_space_available_1k=30092296, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20056, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19312, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42148, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14945008, free=9046040, buffers=844596, cached=7829468, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.22, five_min_avg=3.28, fifteen_min_avg=3.13, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:46:47.025 7f47cc27bc18 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:47:23.869 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50997352, partition_space_available_1k=30093132, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20336, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19384, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42152, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14965348, free=9025656, buffers=844596, cached=7829512, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.32, five_min_avg=3.01, fifteen_min_avg=3.05, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:48:23.873 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51001124, partition_space_available_1k=30089360, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20420, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19440, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42152, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=14997596, free=8988548, buffers=844596, cached=7834372, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.99, five_min_avg=2.75, fifteen_min_avg=2.95, cpu_share=2.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:48:59.252 7f47cc250390 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:49:07.889 7f47cc260828 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:49:19.106 7f47cd6b3cc0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:49:23.892 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51000228, partition_space_available_1k=30090256, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20572, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19504, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42148, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15071820, free=8908648, buffers=844604, cached=7840040, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.13, five_min_avg=2.65, fifteen_min_avg=2.91, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:49:54.513 7f47cd790a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connection Closed 03/31/2025 02:49:54.513 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpClose] in state[Established] 03/31/2025 02:49:59.520 7f47cc27a2e8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:59.520 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[Established] -> State[Connect] 03/31/2025 02:49:59.520 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:59.520 7f47cc27a2e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Established to Connect on EvTcpClose >>, ] description = Collector:Collector[Established to Connect on EvTcpClose] connection down >>, ] >> 03/31/2025 02:49:59.520 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:59.520 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:59.521 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:59.521 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:59.521 7f47cc27a2e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:59.521 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:59.521 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:59.521 7f47cc27a2e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:49:59.521 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:03.523 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:08.526 7f47cc27a2e8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:08.526 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:08.526 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:08.527 7f47cc27a2e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:08.527 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:08.527 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:08.527 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:08.527 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:08.527 7f47cc27a2e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:08.527 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:08.527 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:08.528 7f47cc27a2e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:08.528 7f47cc27a2e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:12.530 7f47cd71b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:17.534 7f47cd71b8d0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:17.534 7f47cd71b8d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:17.534 7f47cd71b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:17.534 7f47cd71b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:17.534 7f47cd71b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:17.534 7f47cd71b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:17.535 7f47cd71b8d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:17.535 7f47cd71b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:17.535 7f47cd71b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:17.535 7f47cd71b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:17.535 7f47cd71b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:17.536 7f47cd71b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:17.536 7f47cd71b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:21.557 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:23.937 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51004600, partition_space_available_1k=30085884, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20776, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19560, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42184, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15408176, free=8561260, buffers=844608, cached=7851068, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.78, five_min_avg=2.93, fifteen_min_avg=2.98, cpu_share=3.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:50:26.572 7f47cc2506d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:26.572 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:26.572 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:26.572 7f47cc2506d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:26.572 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:26.572 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:26.573 7f47cc2506d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51004600 partition_space_available_1k = 30085884 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 20776 cpu_share = 0 mem_res = 0 >>contrail-dns : << mem_virt = 19560 cpu_share = 0 mem_res = 0 >>contrail-named : << mem_virt = 39372 cpu_share = 0 mem_res = 0 >>contrail-control-nodemgr : << mem_virt = 42184 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15408176 free = 8561260 buffers = 844608 cached = 7851068 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.78 five_min_avg = 2.93 fifteen_min_avg = 2.98 cpu_share = 3.25 node_type = control-node >> >> 03/31/2025 02:50:26.573 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:26.573 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:26.573 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:26.573 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:26.573 7f47cc2506d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:26.573 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:26.573 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:26.574 7f47cc2506d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:26.574 7f47cc2506d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:30.575 7f47cd76b4e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:35.581 7f47cd76b4e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:35.581 7f47cd76b4e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:35.581 7f47cd76b4e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:35.582 7f47cd76b4e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:35.582 7f47cd76b4e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:35.582 7f47cd76b4e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:35.582 7f47cd76b4e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:35.582 7f47cd76b4e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:35.582 7f47cd76b4e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:35.582 7f47cd76b4e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:35.582 7f47cd76b4e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:35.583 7f47cd76b4e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:35.583 7f47cd76b4e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:39.582 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:44.588 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:44.588 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:44.589 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:44.589 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:44.589 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:44.589 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:44.589 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:44.589 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:44.589 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:44.590 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:44.590 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:44.590 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:44.590 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:48.085 7f47d0e63ac8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:50:48.589 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:53.591 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:53.592 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:53.592 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:53.592 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:53.592 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:53.592 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:53.593 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:53.593 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:53.593 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:53.593 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:53.593 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:53.593 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:53.593 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:57.593 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:02.598 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:02.599 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:02.599 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:02.599 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:02.599 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:02.599 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:02.600 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:02.600 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:02.600 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:02.600 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:02.600 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:02.600 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:02.600 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:06.603 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:11.608 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:11.608 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:11.609 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:11.609 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:11.609 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:11.609 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:11.609 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:11.609 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:11.610 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:11.610 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:11.610 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:11.610 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:11.610 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:15.611 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:20.614 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:20.615 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:20.615 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:20.615 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:20.615 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:20.615 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:20.616 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:20.616 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:20.616 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:20.616 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:20.617 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:20.617 7f47d0e63ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:20.618 7f47d0e63ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:23.877 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51014644, partition_space_available_1k=30075840, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20920, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19632, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42184, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15012988, free=8948612, buffers=844612, cached=7858900, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.82, five_min_avg=2.89, fifteen_min_avg=2.97, cpu_share=3.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:51:23.877 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:23.877 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51014644 partition_space_available_1k = 30075840 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 20920 cpu_share = 0 mem_res = 0 >>contrail-dns : << mem_virt = 19632 cpu_share = 0 mem_res = 0 >>contrail-named : << mem_virt = 39372 cpu_share = 0 mem_res = 0 >>contrail-control-nodemgr : << mem_virt = 42184 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15012988 free = 8948612 buffers = 844612 cached = 7858900 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.82 five_min_avg = 2.89 fifteen_min_avg = 2.97 cpu_share = 3.38 node_type = control-node >> >> 03/31/2025 02:51:23.878 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:24.616 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:29.622 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:29.622 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:29.622 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:29.622 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:29.622 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:29.622 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:29.623 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:29.623 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:29.623 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:29.623 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:29.623 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:29.623 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:29.623 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:33.625 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:38.628 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:38.628 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:38.628 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:38.628 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:38.628 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:38.629 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:38.629 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:38.629 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:38.629 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:38.629 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:38.629 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:38.629 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:38.629 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:42.632 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:47.637 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:47.637 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:47.637 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:47.637 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:47.637 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:47.638 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:47.638 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:47.638 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:47.638 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:47.638 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:47.638 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:47.639 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:47.639 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:51.641 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:56.645 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:56.646 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:56.646 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:56.646 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:56.646 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:56.646 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:56.647 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:56.647 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:56.647 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:56.647 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:56.647 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:56.647 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:56.648 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:00.649 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:05.653 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:05.653 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:05.654 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:05.654 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:05.654 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:05.654 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:05.654 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:05.654 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:05.654 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:05.655 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:05.655 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:05.655 7f47cd7f7d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:05.655 7f47cd7f7d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:09.655 7f47cf7b6ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:14.661 7f47cf7b6ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:14.662 7f47cf7b6ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:14.662 7f47cf7b6ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:14.662 7f47cf7b6ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:14.662 7f47cf7b6ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:14.662 7f47cf7b6ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:14.663 7f47cf7b6ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:14.663 7f47cf7b6ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:14.663 7f47cf7b6ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:14.663 7f47cf7b6ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:14.663 7f47cf7b6ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:14.663 7f47cf7b6ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:14.663 7f47cf7b6ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:18.663 7f47cd78b4e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:23.665 7f47cd78b4e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:23.666 7f47cd78b4e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:23.666 7f47cd78b4e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:23.666 7f47cd78b4e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:23.666 7f47cd78b4e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:23.666 7f47cd78b4e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:23.667 7f47cd78b4e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:23.667 7f47cd78b4e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:23.667 7f47cd78b4e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:23.667 7f47cd78b4e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:23.667 7f47cd78b4e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:23.667 7f47cd78b4e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:23.667 7f47cd78b4e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:23.884 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51014332, partition_space_available_1k=30076152, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20916, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19696, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42204, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15027796, free=8930400, buffers=844612, cached=7862304, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.36, five_min_avg=2.73, fifteen_min_avg=2.91, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:52:23.885 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:23.885 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51014332 partition_space_available_1k = 30076152 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 20916 cpu_share = 0 mem_res = 0 >>contrail-dns : << mem_virt = 19696 cpu_share = 0 mem_res = 0 >>contrail-named : << mem_virt = 39372 cpu_share = 0 mem_res = 0 >>contrail-control-nodemgr : << mem_virt = 42204 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15027796 free = 8930400 buffers = 844612 cached = 7862304 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.36 five_min_avg = 2.73 fifteen_min_avg = 2.91 cpu_share = 2.62 node_type = control-node >> >> 03/31/2025 02:52:23.885 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:27.669 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:32.675 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:32.675 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:32.675 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:32.676 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:32.676 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:32.676 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:32.676 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:32.677 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:32.677 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:32.677 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:32.677 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:32.677 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:32.677 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:36.679 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:41.683 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:41.684 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:41.684 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:41.684 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:41.684 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:41.684 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:41.684 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:41.684 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:41.685 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:41.685 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:41.685 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:41.685 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:41.685 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:45.688 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:50.693 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:50.693 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:50.693 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:50.693 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:50.694 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:50.694 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:50.694 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:50.694 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:50.694 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:50.694 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:50.694 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:50.694 7f47d1208c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:50.694 7f47d1208c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:54.694 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:59.700 7f47cd7fd780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:59.700 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:59.700 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:59.700 7f47cd7fd780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:59.700 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:59.700 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:59.701 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:59.701 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:59.701 7f47cd7fd780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:59.701 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:59.701 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:59.701 7f47cd7fd780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:59.701 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:03.703 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:08.709 7f47cd7fd780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:08.709 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:08.709 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:08.710 7f47cd7fd780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:08.710 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:08.710 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:08.710 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:08.711 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:08.711 7f47cd7fd780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:08.711 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:08.711 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:08.711 7f47cd7fd780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:08.711 7f47cd7fd780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:12.713 7f47cd6b74e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:17.715 7f47cd6b74e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:17.716 7f47cd6b74e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:17.716 7f47cd6b74e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:17.716 7f47cd6b74e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:17.716 7f47cd6b74e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:17.716 7f47cd6b74e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:17.717 7f47cd6b74e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:17.717 7f47cd6b74e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:17.717 7f47cd6b74e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:17.717 7f47cd6b74e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:17.717 7f47cd6b74e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:17.718 7f47cd6b74e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:17.718 7f47cd6b74e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:21.717 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:23.891 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51017924, partition_space_available_1k=30072560, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21096, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19828, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42256, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15015520, free=8938320, buffers=844612, cached=7866660, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.79, five_min_avg=2.48, fifteen_min_avg=2.81, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:53:26.722 7f47cd77d780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:26.722 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:26.722 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:26.722 7f47cd77d780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:26.722 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:26.722 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:26.723 7f47cd77d780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51017924 partition_space_available_1k = 30072560 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 21096 cpu_share = 0 mem_res = 0 >>contrail-dns : << mem_virt = 19828 cpu_share = 0 mem_res = 0 >>contrail-named : << mem_virt = 39372 cpu_share = 0 mem_res = 0 >>contrail-control-nodemgr : << mem_virt = 42256 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15015520 free = 8938320 buffers = 844612 cached = 7866660 node_type = control-node >> system_cpu_usage = << one_min_avg = 1.79 five_min_avg = 2.48 fifteen_min_avg = 2.81 cpu_share = 2.5 node_type = control-node >> >> 03/31/2025 02:53:26.723 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:26.723 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:26.723 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:26.724 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:26.725 7f47cd77d780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:26.725 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:26.725 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:26.725 7f47cd77d780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:26.725 7f47cd77d780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:30.726 7f47cd6bbc18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:35.732 7f47cd6bbc18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:35.733 7f47cd6bbc18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:35.733 7f47cd6bbc18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:35.733 7f47cd6bbc18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:35.733 7f47cd6bbc18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:35.733 7f47cd6bbc18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:35.733 7f47cd6bbc18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:35.733 7f47cd6bbc18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:35.734 7f47cd6bbc18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:35.734 7f47cd6bbc18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:35.734 7f47cd6bbc18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:35.734 7f47cd6bbc18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:35.734 7f47cd6bbc18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:39.733 7f47cd790780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:44.737 7f47cd790780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:44.737 7f47cd790780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:44.737 7f47cd790780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:44.737 7f47cd790780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:44.737 7f47cd790780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:44.737 7f47cd790780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:44.738 7f47cd790780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:44.738 7f47cd790780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:44.738 7f47cd790780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:44.738 7f47cd790780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:44.738 7f47cd790780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:44.738 7f47cd790780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:44.738 7f47cd790780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:48.740 7f47cd790780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:53.746 7f47cd790780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:53.747 7f47cd790780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:53.747 7f47cd790780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:53.747 7f47cd790780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:53.747 7f47cd790780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:53.747 7f47cd790780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:53.747 7f47cd790780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:53.748 7f47cd790780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:53.748 7f47cd790780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:53.748 7f47cd790780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:53.748 7f47cd790780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:53.748 7f47cd790780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:53.748 7f47cd790780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:57.749 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:02.755 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:02.755 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:02.755 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:02.755 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:02.755 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:02.755 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:02.756 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:02.756 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:02.756 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:02.756 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:02.756 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:02.756 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:02.756 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:06.758 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:11.761 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:11.762 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:11.762 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:11.762 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:11.762 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:11.762 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:11.762 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:11.763 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:11.763 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:11.763 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:11.763 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:11.763 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:11.763 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:12.217 7f47cfdc14e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:54:15.765 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:20.767 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:20.767 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:20.767 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:20.767 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:20.768 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:20.768 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:20.768 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:20.768 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:20.768 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:20.769 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:20.769 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:20.769 7f47cfdc14e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:20.769 7f47cfdc14e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:20.829 7f47cfdc14e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:54:23.905 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51021456, partition_space_available_1k=30069028, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21224, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19884, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42248, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15090000, free=8859768, buffers=844612, cached=7870732, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.04, five_min_avg=2.41, fifteen_min_avg=2.76, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:54:23.906 7f47cd6da240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:23.906 7f47cd6da240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51021456 partition_space_available_1k = 30069028 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 21224 cpu_share = 0 mem_res = 0 >>contrail-dns : << mem_virt = 19884 cpu_share = 0 mem_res = 0 >>contrail-named : << mem_virt = 39372 cpu_share = 0 mem_res = 0 >>contrail-control-nodemgr : << mem_virt = 42248 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15090000 free = 8859768 buffers = 844612 cached = 7870732 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.04 five_min_avg = 2.41 fifteen_min_avg = 2.76 cpu_share = 2.62 node_type = control-node >> >> 03/31/2025 02:54:23.906 7f47cd6da240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:24.767 7f47cd7832e8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:29.773 7f47cd7832e8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:29.773 7f47cd7832e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:29.773 7f47cd7832e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:29.773 7f47cd7832e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:29.773 7f47cd7832e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:29.774 7f47cd7832e8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:29.774 7f47cd7832e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:29.774 7f47cd7832e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:29.774 7f47cd7832e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:29.774 7f47cd7832e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:29.774 7f47cd7832e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:29.775 7f47cd7832e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:29.775 7f47cd7832e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:32.031 7f47cd7832e8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:54:33.774 7f47cc3eec18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:38.777 7f47cc3eec18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:38.777 7f47cc3eec18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:38.777 7f47cc3eec18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:38.777 7f47cc3eec18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:38.777 7f47cc3eec18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:38.777 7f47cc3eec18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:38.778 7f47cc3eec18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:38.778 7f47cc3eec18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:38.778 7f47cc3eec18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:38.778 7f47cc3eec18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:38.778 7f47cc3eec18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:38.778 7f47cc3eec18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:38.779 7f47cc3eec18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:42.780 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:47.786 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:47.787 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:47.787 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:47.787 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:47.787 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:47.787 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:47.787 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:47.787 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:47.788 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:47.788 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:47.788 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:47.788 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:47.788 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:51.790 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:56.793 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:56.793 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:56.793 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:56.793 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:56.793 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:56.793 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:56.794 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:56.794 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:56.794 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:56.794 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:56.794 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:56.794 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:56.794 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:00.796 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:05.802 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:05.802 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:05.802 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:05.803 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:05.803 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:05.803 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:05.803 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:05.803 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:05.803 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:05.803 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:05.803 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:05.804 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:05.804 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:09.803 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:14.809 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:14.809 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:14.809 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:14.810 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:14.810 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:14.810 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:14.810 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:14.810 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:14.810 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:14.810 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:14.811 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:14.811 7f47cd77df60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:14.811 7f47cd77df60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:18.810 7f47d1259588 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:23.815 7f47d1259588 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:23.815 7f47d1259588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:23.815 7f47d1259588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:23.815 7f47d1259588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:23.815 7f47d1259588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:23.815 7f47d1259588 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:23.816 7f47d1259588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:23.816 7f47d1259588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:23.816 7f47d1259588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:23.816 7f47d1259588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:23.816 7f47d1259588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:23.816 7f47d1259588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:23.816 7f47d1259588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:23.898 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51021928, partition_space_available_1k=30068556, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21420, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19952, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42272, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15036580, free=8910916, buffers=844612, cached=7873004, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.78, five_min_avg=2.27, fifteen_min_avg=2.69, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:55:23.898 7f47cd77d0f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:23.899 7f47cd77d0f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51021928 partition_space_available_1k = 30068556 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 21420 cpu_share = 0 mem_res = 0 >>contrail-dns : << mem_virt = 19952 cpu_share = 0 mem_res = 0 >>contrail-named : << mem_virt = 39372 cpu_share = 0 mem_res = 0 >>contrail-control-nodemgr : << mem_virt = 42272 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15036580 free = 8910916 buffers = 844612 cached = 7873004 node_type = control-node >> system_cpu_usage = << one_min_avg = 1.78 five_min_avg = 2.27 fifteen_min_avg = 2.69 cpu_share = 2.62 node_type = control-node >> >> 03/31/2025 02:55:23.899 7f47cd77d0f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:27.817 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:32.822 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:32.823 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:32.823 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:32.823 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:32.823 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:32.823 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:32.823 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:32.823 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:32.824 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:32.824 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:32.824 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:32.824 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:32.824 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:36.826 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:41.845 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:41.845 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:41.845 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:41.845 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:41.845 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:41.845 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:41.846 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:41.846 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:41.846 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:41.846 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:41.846 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:41.846 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:41.846 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:45.849 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:50.855 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:50.855 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:50.855 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:50.855 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:50.855 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:50.855 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:50.856 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:50.856 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:50.856 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:50.856 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:50.856 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:50.856 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:50.856 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:54.858 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:59.861 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:59.861 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:59.861 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:59.861 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:59.861 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:59.861 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:59.862 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:59.862 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:59.862 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:59.862 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:59.862 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:59.862 7f47cd78a240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:59.862 7f47cd78a240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:03.863 7f47cfb8eb70 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:08.869 7f47cfb8eb70 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:08.870 7f47cfb8eb70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:08.870 7f47cfb8eb70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:08.870 7f47cfb8eb70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:08.870 7f47cfb8eb70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:08.870 7f47cfb8eb70 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:08.870 7f47cfb8eb70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:08.870 7f47cfb8eb70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:08.871 7f47cfb8eb70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:08.871 7f47cfb8eb70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:08.871 7f47cfb8eb70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:08.871 7f47cfb8eb70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:08.871 7f47cfb8eb70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:12.872 7f47cd740f60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:17.873 7f47cd740f60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:17.874 7f47cd740f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:17.874 7f47cd740f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:17.874 7f47cd740f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:17.874 7f47cd740f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:17.874 7f47cd740f60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:17.875 7f47cd740f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:17.875 7f47cd740f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:17.875 7f47cd740f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:17.875 7f47cd740f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:17.875 7f47cd740f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:17.875 7f47cd740f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:17.875 7f47cd740f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:21.875 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:23.891 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51026248, partition_space_available_1k=30064236, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21644, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20016, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42332, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15052660, free=8876580, buffers=844620, cached=7891252, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.79, five_min_avg=3.23, fifteen_min_avg=3.0, cpu_share=3.88, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:56:26.878 7f47cc251d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:26.879 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:26.879 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:26.879 7f47cc251d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:26.879 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:26.879 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:26.879 7f47cc251d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51026248 partition_space_available_1k = 30064236 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 21644 cpu_share = 0 mem_res = 0 >>contrail-dns : << mem_virt = 20016 cpu_share = 0 mem_res = 0 >>contrail-named : << mem_virt = 39372 cpu_share = 0 mem_res = 0 >>contrail-control-nodemgr : << mem_virt = 42332 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15052660 free = 8876580 buffers = 844620 cached = 7891252 node_type = control-node >> system_cpu_usage = << one_min_avg = 4.79 five_min_avg = 3.23 fifteen_min_avg = 3.0 cpu_share = 3.88 node_type = control-node >> >> 03/31/2025 02:56:26.879 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:26.879 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:26.880 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:26.880 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:26.880 7f47cc251d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:26.880 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:26.880 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:26.880 7f47cc251d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:26.880 7f47cc251d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:30.882 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:35.888 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:35.889 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:35.889 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:35.889 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:35.889 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:35.889 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:35.889 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:35.890 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:35.890 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:35.890 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:35.890 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:35.890 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:35.890 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:37.457 7f47cfd56198 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:56:39.890 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:44.895 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:44.895 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:44.895 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:44.895 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:44.895 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:44.895 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:44.896 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:44.896 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:44.896 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:44.896 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:44.896 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:44.896 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:44.896 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:48.899 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:53.905 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:53.905 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:53.906 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:53.906 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:53.906 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:53.906 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:53.907 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:53.907 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:53.907 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:53.907 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:53.907 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:53.907 7f47cfd56198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:53.907 7f47cfd56198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:57.906 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:02.912 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:02.912 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:02.912 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:02.912 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:02.913 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:02.913 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:02.913 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:02.913 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:02.913 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:02.913 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:02.913 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:02.914 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:57:02.914 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:06.915 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:11.917 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:11.918 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:11.918 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:11.918 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:11.918 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:11.918 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:11.918 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:11.919 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:11.919 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:11.919 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:11.919 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:11.919 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:57:11.919 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:15.922 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:20.924 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:20.924 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:20.925 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:20.925 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:20.925 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:20.925 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:20.926 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:20.926 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:20.926 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:20.926 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:20.926 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:20.926 7f47cfd51588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:57:20.926 7f47cfd51588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:23.892 7f47cfd5cd48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-494-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51031240, partition_space_available_1k=30059244, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21708, cpu_share=0, mem_res=0, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20136, cpu_share=0, mem_res=0, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39372, cpu_share=0, mem_res=0, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42320, cpu_share=0, mem_res=0, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=15045208, free=8876256, buffers=844620, cached=7899028, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.71, five_min_avg=2.91, fifteen_min_avg=2.9, cpu_share=2.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 03/31/2025 02:57:23.892 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:23.892 7f47cd6b7eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51031240 partition_space_available_1k = 30059244 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 21708 cpu_share = 0 mem_res = 0 >>contrail-dns : << mem_virt = 20136 cpu_share = 0 mem_res = 0 >>contrail-named : << mem_virt = 39372 cpu_share = 0 mem_res = 0 >>contrail-control-nodemgr : << mem_virt = 42320 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665112 used = 15045208 free = 8876256 buffers = 844620 cached = 7899028 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.71 five_min_avg = 2.91 fifteen_min_avg = 2.9 cpu_share = 2.25 node_type = control-node >> >> 03/31/2025 02:57:23.893 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:24.926 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:29.931 7f47cd6b7eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:29.932 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:29.932 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:29.932 7f47cd6b7eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:29.932 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:29.932 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:29.933 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:29.933 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:29.933 7f47cd6b7eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:29.933 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:29.933 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:29.933 7f47cd6b7eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:57:29.933 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:32.353 7f47cd6b7eb8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 03/31/2025 02:57:33.933 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:38.939 7f47cd6b7eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:38.939 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:38.939 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:38.939 7f47cd6b7eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:38.939 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:38.939 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:38.940 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:38.940 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:38.940 7f47cd6b7eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:38.940 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:38.940 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:38.940 7f47cd6b7eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8101 start_time = 1743387255572633 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:57:38.940 7f47cd6b7eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]