INFO: =================== Mon Aug 18 01:39:50 UTC 2025 =================== /contrail-functions.sh: line 167: /etc/contrail/vnc_api_lib.ini.tmp: No such file or directory /contrail-functions.sh: line 202: /etc/contrail/vnc_api_lib.ini.tmp: No such file or directory mv: cannot stat ‘/etc/contrail/vnc_api_lib.ini.tmp’: No such file or directory [DEFAULTS] http_server_ip=0.0.0.0 log_file=/var/log/contrail/database-nodemgr/contrail-database-nodemgr.log log_level=SYS_NOTICE log_local=1 hostip=10.0.0.49 db_port=9042 db_jmx_port=7200 db_use_ssl=False [COLLECTOR] server_list=10.0.0.49:8086 [SANDESH] introspect_ssl_enable=False sandesh_ssl_enable=False [DEFAULTS] minimum_diskGB = 2 08/18/2025 01:39:51.639 7f5979a7ca88 [contrail-database-nodemgr] [INFO]: SANDESH: CONNECT TO COLLECTOR: True 08/18/2025 01:39:51.757 7f5979a7ca88 [contrail-database-nodemgr] [ERROR]: Failed to import package "database.sandesh" 08/18/2025 01:39:51.765 7f5979a7ca88 [contrail-database-nodemgr] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8103 08/18/2025 01:39:51.769 7f5979a7ca88 [contrail-database-nodemgr] [ERROR]: Failed to import package "database.sandesh" 08/18/2025 01:39:51.776 7f5979a7ca88 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_NOTICE] CRITICAL:root:Exit data: {'ID': 'e28787e4f006e8fde39ab1b95ffb5f47d48e559b75ac03b35cbc76ed8c7fc773', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.49', '9042', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 76339} ERROR:contrail-database-nodemgr:Failed to connect to database by CQL and socket: Result: 1 CRITICAL:root:Exit data: {'ID': '4e8e1892607d32ebfd11e36eb1a755eeb15cfcea2fb5c9e39f882ed815550060', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7200', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 76377} ERROR:contrail-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' [Introspect:8103]10.0.0.49 - - [2025-08-18 01:41:51] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6690 0.001467 CRITICAL:root:Exit data: {'ID': '8f5094ac55c290471d789a278195b4067a03626a10084bbb40a3b24723d759e9', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7200', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 78238} ERROR:contrail-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' [Introspect:8103]10.0.0.49 - - [2025-08-18 01:42:04] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6656 0.001413 [Introspect:8103]10.0.0.49 - - [2025-08-18 01:42:17] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6656 0.001190 [Introspect:8103]10.0.0.49 - - [2025-08-18 01:42:20] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6656 0.001261 ERROR:root:docker Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/e939b0622e34464ec0b9fc740e94f7ad49243db7426e2ed15972e8f48b0216fe/json During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/nodemgr/common/docker_containers.py", line 17, in inspect return self._client.inspect_container(id_) File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped return f(self, resource_id, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/docker/api/container.py", line 772, in inspect_container self._get(self._url("/containers/{0}/json", container)), True File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 274, in _result self._raise_for_status(response) File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) File "/usr/local/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/e939b0622e34464ec0b9fc740e94f7ad49243db7426e2ed15972e8f48b0216fe/json: Not Found ("No such container: e939b0622e34464ec0b9fc740e94f7ad49243db7426e2ed15972e8f48b0216fe") CRITICAL:root:Exit data: {'ID': '6e99892741297e55c60a14a841145ebd6f2d4c2a05d8df8fbf311cdf62924ee3', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7200', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 80369} ERROR:contrail-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '797dfb2eb8dffc7ebee3f6e14f133010cae3dcc613b208f840b335595245aae7', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7200', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 83299} ERROR:contrail-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '9b16273d044779ddf05640848118eeeacdc1ff2794d4ce1afa335b30ab809872', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7200', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 86037} ERROR:contrail-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '1e7c5667d6e6326d0278b2e5fefe4511976406a6b195af8d3a854b06e3a36336', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7200', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 87670} ERROR:contrail-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '367e5d2e856224db77ebd839ee3e4e1ad7b52602ed35e78dd62d5f6210b3e781', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7200', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 88764} ERROR:contrail-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '5e175995f1ae55d1e383946b241c3391a4b73828c4af2a4f0d361a319fd6c937', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7200', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 89851} ERROR:contrail-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': 'e54e185ed76530ac5eedecd153e30477eb12014827bc2c0ef315d0f1ff3ad60a', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7200', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 91803} ERROR:contrail-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' [Introspect:8103]10.0.0.49 - - [2025-08-18 01:49:35] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6656 0.001311 CRITICAL:root:Exit data: {'ID': 'b564ff8efca867fb00849adf8cc7b2a3afaac148121285fcdb7b9f03ab35e435', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7200', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'd968a26e05d9ba54f6addb2d069068bcaf3e00dcb9f61e956dfaa955824ebd07', 'DetachKeys': '', 'Pid': 97067} ERROR:contrail-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'