2025-10-20 05:03:55.935 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:03:58.893 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:03:58.893 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:03:58.920 [info] <0.240.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:03:58.920 [info] <0.240.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:03:58.934 [info] <0.240.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:03:58.934 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:03:58.934 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:03:58.939 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:03:58.954 [info] <0.248.0> Enabling free disk space monitoring 2025-10-20 05:03:58.954 [info] <0.248.0> Disk free limit set to 50MB 2025-10-20 05:03:58.961 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:03:58.961 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:03:58.962 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:03:58.962 [info] <0.252.0> FHC read buffering: OFF 2025-10-20 05:03:58.962 [info] <0.252.0> FHC write buffering: ON 2025-10-20 05:03:58.962 [info] <0.240.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:03:58.962 [info] <0.241.0> Will use 4 processes for default worker pool 2025-10-20 05:03:58.962 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:03:58.962 [info] <0.240.0> Running boot step database defined by app rabbit 2025-10-20 05:03:58.962 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:03:58.962 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:03:58.962 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:03:58.962 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:03:58.962 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:03:58.962 [info] <0.240.0> Will wait for 50450 milliseconds before proceeding with registration... 2025-10-20 05:04:41.376 [info] <0.294.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping... 2025-10-20 05:04:49.413 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:04:49.413 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:04:49.417 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:04:49.420 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:04:49.420 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:04:49.922 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:04:49.923 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:04:49.923 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:04:50.424 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:04:50.425 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:04:50.425 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:04:50.926 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:04:50.927 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:04:50.927 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:04:51.429 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:04:51.432 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:04:51.432 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:04:51.933 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:04:51.934 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:04:51.934 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:04:52.436 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:04:52.437 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:04:52.437 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:04:52.939 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:04:52.940 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:04:52.940 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:04:53.445 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:04:53.445 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:04:53.446 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:04:53.947 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:04:53.947 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:04:53.947 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:04:54.448 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:04:54.450 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:04:54.569 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:04:54.569 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:04:54.600 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:04:54.600 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:04:54.636 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:04:54.636 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:04:54.636 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:04:54.636 [info] <0.240.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:04:54.636 [info] <0.240.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:04:54.636 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:04:54.636 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:04:54.636 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:04:54.637 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:04:54.638 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:04:54.638 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:04:54.638 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:04:54.638 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:04:54.638 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:04:54.638 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:04:54.638 [info] <0.471.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:04:54.639 [info] <0.240.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:04:54.643 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:04:54.643 [info] <0.475.0> Starting rabbit_node_monitor 2025-10-20 05:04:54.643 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:04:54.643 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:04:54.644 [info] <0.240.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:04:54.644 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:04:54.665 [info] <0.240.0> message_store upgrades: 1 to apply 2025-10-20 05:04:54.665 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:04:54.665 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:04:54.665 [info] <0.240.0> message_store upgrades: Removing the old message store data 2025-10-20 05:04:54.666 [info] <0.240.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:04:54.690 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:04:54.690 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:04:54.690 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:04:54.690 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:04:54.690 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:04:54.691 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:04:54.691 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:04:54.691 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:04:54.691 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:04:54.691 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:04:54.691 [warning] <0.240.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:04:54.691 [info] <0.240.0> Management plugin: using rates mode 'basic' 2025-10-20 05:04:54.692 [info] <0.240.0> Running boot step recovery defined by app rabbit 2025-10-20 05:04:54.692 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:04:54.692 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:04:54.692 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:04:54.692 [info] <0.240.0> Importing users... 2025-10-20 05:04:54.693 [info] <0.240.0> Creating user 'guest' 2025-10-20 05:04:54.699 [info] <0.240.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:04:54.703 [info] <0.240.0> Importing vhosts... 2025-10-20 05:04:54.703 [info] <0.240.0> Adding vhost '/' 2025-10-20 05:04:54.727 [info] <0.523.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:04:54.736 [info] <0.523.0> Starting message stores for vhost '/' 2025-10-20 05:04:54.737 [info] <0.527.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:04:54.740 [info] <0.523.0> Started message store of type transient for vhost '/' 2025-10-20 05:04:54.740 [info] <0.530.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:04:54.742 [warning] <0.530.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:04:54.744 [info] <0.523.0> Started message store of type persistent for vhost '/' 2025-10-20 05:04:54.745 [info] <0.240.0> Importing user permissions... 2025-10-20 05:04:54.745 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:04:54.749 [info] <0.240.0> Importing topic permissions... 2025-10-20 05:04:54.749 [info] <0.240.0> Importing parameters... 2025-10-20 05:04:54.749 [info] <0.240.0> Importing global parameters... 2025-10-20 05:04:54.750 [info] <0.240.0> Importing policies... 2025-10-20 05:04:54.759 [info] <0.240.0> Importing queues... 2025-10-20 05:04:54.759 [info] <0.240.0> Importing exchanges... 2025-10-20 05:04:54.759 [info] <0.240.0> Importing bindings... 2025-10-20 05:04:54.760 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:04:54.760 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:04:54.760 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:04:54.760 [info] <0.240.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:04:54.760 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:04:54.764 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:04:54.773 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:04:54.773 [info] <0.240.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:04:54.773 [info] <0.240.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:04:54.773 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:04:54.773 [info] <0.240.0> Running boot step networking defined by app rabbit 2025-10-20 05:04:54.777 [info] <0.576.0> started TCP listener on [::]:5673 2025-10-20 05:04:54.777 [info] <0.240.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:04:54.777 [info] <0.240.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:04:54.778 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:04:54.778 [info] <0.578.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:04:54.806 [info] <0.628.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:04:54.806 [info] <0.734.0> Statistics database started. 2025-10-20 05:04:54.806 [info] <0.733.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:04:54.807 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:04:55.000 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_web_dispatch * rabbitmq_management_agent 2025-10-20 05:04:55.013 [info] <0.294.0> RabbitMQ is asked to stop... 2025-10-20 05:04:55.099 [info] <0.294.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:04:55.099 [info] <0.294.0> Stopping application 'rabbitmq_management' 2025-10-20 05:04:55.100 [warning] <0.624.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:04:55.102 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:04:55.102 [info] <0.294.0> Stopping application 'amqp_client' 2025-10-20 05:04:55.103 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:04:55.103 [info] <0.294.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:04:55.104 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:04:55.104 [info] <0.294.0> Stopping application 'cowboy' 2025-10-20 05:04:55.106 [info] <0.294.0> Stopping application 'cowlib' 2025-10-20 05:04:55.106 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:04:55.106 [info] <0.294.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:04:55.106 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:04:55.107 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:04:55.107 [info] <0.294.0> Stopping application 'rabbit' 2025-10-20 05:04:55.107 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:04:55.108 [info] <0.576.0> stopped TCP listener on [::]:5673 2025-10-20 05:04:55.108 [info] <0.500.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:04:55.108 [info] <0.530.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' 2025-10-20 05:04:55.114 [info] <0.530.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped 2025-10-20 05:04:55.114 [info] <0.527.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' 2025-10-20 05:04:55.120 [info] <0.527.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped 2025-10-20 05:04:55.123 [info] <0.294.0> Stopping application 'mnesia' 2025-10-20 05:04:55.123 [info] <0.43.0> Application rabbit exited with reason: stopped 2025-10-20 05:04:55.129 [info] <0.294.0> Stopping application 'rabbit_common' 2025-10-20 05:04:55.129 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:04:55.129 [info] <0.43.0> Application rabbit_common exited with reason: stopped 2025-10-20 05:04:55.129 [info] <0.294.0> Stopping application 'sysmon_handler' 2025-10-20 05:04:55.130 [info] <0.294.0> Stopping application 'os_mon' 2025-10-20 05:04:55.130 [info] <0.43.0> Application sysmon_handler exited with reason: stopped 2025-10-20 05:04:55.131 [info] <0.294.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:04:55.131 [info] <0.43.0> Application os_mon exited with reason: stopped 2025-10-20 05:05:04.779 [info] <0.877.0> Resetting Rabbit forcefully 2025-10-20 05:05:06.531 [info] <0.889.0> RabbitMQ is asked to stop... 2025-10-20 05:05:06.579 [info] <0.889.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:05:06.579 [info] <0.889.0> Stopping application 'rabbitmq_management' 2025-10-20 05:05:06.579 [info] <0.889.0> Stopping application 'amqp_client' 2025-10-20 05:05:06.579 [info] <0.889.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:05:06.579 [info] <0.889.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:05:06.579 [info] <0.889.0> Stopping application 'rabbit' 2025-10-20 05:05:06.579 [info] <0.889.0> Stopping application 'mnesia' 2025-10-20 05:05:06.579 [info] <0.889.0> Stopping application 'rabbit_common' 2025-10-20 05:05:06.579 [info] <0.889.0> Stopping application 'sysmon_handler' 2025-10-20 05:05:06.579 [info] <0.889.0> Stopping application 'os_mon' 2025-10-20 05:05:06.579 [info] <0.889.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:05:06.579 [info] <0.889.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 observer_cli tools credentials_obfuscation crypto stdout_formatter recon inets jsx xmerl lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:05:11.677 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:05:13.113 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:05:13.114 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:05:13.136 [info] <0.240.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:05:13.136 [info] <0.240.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:05:13.148 [info] <0.240.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:05:13.148 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:05:13.148 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:05:13.153 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:05:13.157 [info] <0.248.0> Enabling free disk space monitoring 2025-10-20 05:05:13.157 [info] <0.248.0> Disk free limit set to 50MB 2025-10-20 05:05:13.160 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:05:13.160 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:05:13.161 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:05:13.161 [info] <0.252.0> FHC read buffering: OFF 2025-10-20 05:05:13.161 [info] <0.252.0> FHC write buffering: ON 2025-10-20 05:05:13.161 [info] <0.240.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:05:13.161 [info] <0.241.0> Will use 4 processes for default worker pool 2025-10-20 05:05:13.161 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:05:13.161 [info] <0.240.0> Running boot step database defined by app rabbit 2025-10-20 05:05:13.161 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:05:13.161 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:05:13.161 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:05:13.161 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:05:13.161 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:05:13.162 [info] <0.240.0> Will wait for 29172 milliseconds before proceeding with registration... 2025-10-20 05:05:42.334 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:05:42.334 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:05:42.345 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:05:42.349 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:05:42.349 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:05:42.853 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:05:42.854 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:05:42.854 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:05:43.356 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:05:43.356 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:05:43.357 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:05:43.858 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:05:43.858 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:05:43.859 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:05:44.366 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:05:44.367 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:05:44.367 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:05:44.868 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:05:44.868 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:05:44.868 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:05:45.370 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:05:45.371 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:05:45.371 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:05:45.874 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:05:45.875 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:05:45.875 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:05:46.384 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:05:46.385 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:05:46.385 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:05:46.886 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:05:46.887 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:05:46.887 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:05:47.388 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:05:47.390 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:05:47.758 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:05:47.758 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:05:47.783 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:05:47.783 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:05:47.812 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:05:47.812 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:05:47.812 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:05:47.812 [info] <0.240.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:05:47.812 [info] <0.240.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:05:47.812 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:05:47.812 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:05:47.812 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:05:47.812 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:05:47.812 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:05:47.812 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:05:47.812 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:05:47.813 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:05:47.822 [info] <0.453.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:05:47.822 [info] <0.240.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:05:47.888 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:05:47.888 [info] <0.457.0> Starting rabbit_node_monitor 2025-10-20 05:05:47.888 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:05:47.888 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:05:47.889 [info] <0.240.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:05:47.889 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:05:47.918 [info] <0.240.0> message_store upgrades: 1 to apply 2025-10-20 05:05:47.918 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:05:47.918 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:05:47.918 [info] <0.240.0> message_store upgrades: Removing the old message store data 2025-10-20 05:05:47.919 [info] <0.240.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:05:47.983 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:05:47.983 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:05:47.983 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:05:47.984 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:05:47.984 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:05:47.984 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:05:47.984 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:05:47.985 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:05:47.985 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:05:47.985 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:05:47.985 [warning] <0.240.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:05:47.985 [info] <0.240.0> Management plugin: using rates mode 'basic' 2025-10-20 05:05:47.985 [info] <0.240.0> Running boot step recovery defined by app rabbit 2025-10-20 05:05:47.986 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:05:47.987 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:05:47.987 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:05:47.987 [info] <0.240.0> Importing users... 2025-10-20 05:05:47.987 [info] <0.240.0> Creating user 'guest' 2025-10-20 05:05:48.036 [info] <0.240.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:05:48.055 [info] <0.240.0> Importing vhosts... 2025-10-20 05:05:48.055 [info] <0.240.0> Adding vhost '/' 2025-10-20 05:05:48.146 [info] <0.505.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:05:48.156 [info] <0.505.0> Starting message stores for vhost '/' 2025-10-20 05:05:48.157 [info] <0.509.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:05:48.158 [info] <0.505.0> Started message store of type transient for vhost '/' 2025-10-20 05:05:48.158 [info] <0.512.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:05:48.158 [warning] <0.512.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:05:48.159 [info] <0.505.0> Started message store of type persistent for vhost '/' 2025-10-20 05:05:48.160 [info] <0.240.0> Importing user permissions... 2025-10-20 05:05:48.160 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:05:48.165 [info] <0.240.0> Importing topic permissions... 2025-10-20 05:05:48.165 [info] <0.240.0> Importing parameters... 2025-10-20 05:05:48.165 [info] <0.240.0> Importing global parameters... 2025-10-20 05:05:48.165 [info] <0.240.0> Importing policies... 2025-10-20 05:05:48.174 [info] <0.240.0> Importing queues... 2025-10-20 05:05:48.174 [info] <0.240.0> Importing exchanges... 2025-10-20 05:05:48.174 [info] <0.240.0> Importing bindings... 2025-10-20 05:05:48.174 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:05:48.174 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:05:48.174 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:05:48.174 [info] <0.240.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:05:48.174 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:05:48.178 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:05:48.185 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:05:48.185 [info] <0.240.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:05:48.185 [info] <0.240.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:05:48.185 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:05:48.185 [info] <0.240.0> Running boot step networking defined by app rabbit 2025-10-20 05:05:48.186 [info] <0.558.0> started TCP listener on [::]:5673 2025-10-20 05:05:48.189 [info] <0.240.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:05:48.189 [info] <0.240.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:05:48.189 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:05:48.189 [info] <0.560.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:05:48.226 [info] <0.610.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:05:48.226 [info] <0.716.0> Statistics database started. 2025-10-20 05:05:48.226 [info] <0.715.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:05:48.227 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:05:48.323 [info] <0.726.0> accepting AMQP connection <0.726.0> (10.0.0.48:46798 -> 10.0.0.48:5673) 2025-10-20 05:05:48.337 [info] <0.726.0> connection <0.726.0> (10.0.0.48:46798 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:48.361 [info] <0.746.0> accepting AMQP connection <0.746.0> (10.0.0.48:46800 -> 10.0.0.48:5673) 2025-10-20 05:05:48.366 [info] <0.746.0> connection <0.746.0> (10.0.0.48:46800 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:48.384 [info] <0.726.0> closing AMQP connection <0.726.0> (10.0.0.48:46798 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:48.384 [info] <0.746.0> closing AMQP connection <0.746.0> (10.0.0.48:46800 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:48.523 [info] <0.761.0> accepting AMQP connection <0.761.0> (10.0.0.49:56372 -> 10.0.0.48:5673) 2025-10-20 05:05:48.524 [info] <0.764.0> accepting AMQP connection <0.764.0> (10.0.0.49:56386 -> 10.0.0.48:5673) 2025-10-20 05:05:48.534 [info] <0.761.0> connection <0.761.0> (10.0.0.49:56372 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:48.534 [info] <0.764.0> connection <0.764.0> (10.0.0.49:56386 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:48.539 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch 2025-10-20 05:05:48.565 [info] <0.796.0> accepting AMQP connection <0.796.0> (10.0.0.49:56394 -> 10.0.0.48:5673) 2025-10-20 05:05:48.574 [info] <0.801.0> accepting AMQP connection <0.801.0> (10.0.0.49:56404 -> 10.0.0.48:5673) 2025-10-20 05:05:48.574 [info] <0.796.0> connection <0.796.0> (10.0.0.49:56394 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:48.578 [info] <0.801.0> connection <0.801.0> (10.0.0.49:56404 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:48.628 [info] <0.764.0> closing AMQP connection <0.764.0> (10.0.0.49:56386 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:48.629 [info] <0.796.0> closing AMQP connection <0.796.0> (10.0.0.49:56394 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:48.646 [info] <0.761.0> closing AMQP connection <0.761.0> (10.0.0.49:56372 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:48.646 [info] <0.827.0> accepting AMQP connection <0.827.0> (10.0.0.254:40416 -> 10.0.0.48:5673) 2025-10-20 05:05:48.647 [info] <0.801.0> closing AMQP connection <0.801.0> (10.0.0.49:56404 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:48.649 [info] <0.827.0> connection <0.827.0> (10.0.0.254:40416 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:48.676 [info] <0.845.0> accepting AMQP connection <0.845.0> (10.0.0.254:40430 -> 10.0.0.48:5673) 2025-10-20 05:05:48.681 [info] <0.845.0> connection <0.845.0> (10.0.0.254:40430 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:48.696 [info] <0.827.0> closing AMQP connection <0.827.0> (10.0.0.254:40416 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:48.697 [info] <0.845.0> closing AMQP connection <0.845.0> (10.0.0.254:40430 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:48.951 [info] <0.860.0> accepting AMQP connection <0.860.0> (10.0.0.49:56408 -> 10.0.0.48:5673) 2025-10-20 05:05:48.954 [info] <0.860.0> connection <0.860.0> (10.0.0.49:56408 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.059 [info] <0.878.0> accepting AMQP connection <0.878.0> (10.0.0.49:56424 -> 10.0.0.48:5673) 2025-10-20 05:05:49.061 [info] <0.878.0> connection <0.878.0> (10.0.0.49:56424 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.144 [info] <0.896.0> accepting AMQP connection <0.896.0> (10.0.0.48:46814 -> 10.0.0.48:5673) 2025-10-20 05:05:49.145 [info] <0.896.0> connection <0.896.0> (10.0.0.48:46814 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.149 [info] <0.914.0> accepting AMQP connection <0.914.0> (10.0.0.48:46828 -> 10.0.0.48:5673) 2025-10-20 05:05:49.154 [info] <0.914.0> connection <0.914.0> (10.0.0.48:46828 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.224 [info] <0.932.0> accepting AMQP connection <0.932.0> (10.0.0.254:40438 -> 10.0.0.48:5673) 2025-10-20 05:05:49.227 [info] <0.932.0> connection <0.932.0> (10.0.0.254:40438 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.292 [info] <0.951.0> accepting AMQP connection <0.951.0> (10.0.0.254:40450 -> 10.0.0.48:5673) 2025-10-20 05:05:49.298 [info] <0.951.0> connection <0.951.0> (10.0.0.254:40450 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.640 [info] <0.969.0> accepting AMQP connection <0.969.0> (10.0.0.48:46838 -> 10.0.0.48:5673) 2025-10-20 05:05:49.643 [info] <0.969.0> connection <0.969.0> (10.0.0.48:46838 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.658 [info] <0.987.0> accepting AMQP connection <0.987.0> (10.0.0.48:46848 -> 10.0.0.48:5673) 2025-10-20 05:05:49.660 [info] <0.987.0> connection <0.987.0> (10.0.0.48:46848 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.675 [info] <0.969.0> closing AMQP connection <0.969.0> (10.0.0.48:46838 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:49.679 [info] <0.987.0> closing AMQP connection <0.987.0> (10.0.0.48:46848 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:49.681 [info] <0.1002.0> accepting AMQP connection <0.1002.0> (10.0.0.48:46852 -> 10.0.0.48:5673) 2025-10-20 05:05:49.682 [info] <0.1005.0> accepting AMQP connection <0.1005.0> (10.0.0.48:46854 -> 10.0.0.48:5673) 2025-10-20 05:05:49.684 [info] <0.1002.0> connection <0.1002.0> (10.0.0.48:46852 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.690 [info] <0.1005.0> connection <0.1005.0> (10.0.0.48:46854 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.916 [info] <0.1002.0> closing AMQP connection <0.1002.0> (10.0.0.48:46852 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:49.917 [info] <0.1012.0> accepting AMQP connection <0.1012.0> (10.0.0.48:46864 -> 10.0.0.48:5673) 2025-10-20 05:05:49.919 [info] <0.1012.0> connection <0.1012.0> (10.0.0.48:46864 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:49.937 [info] <0.1012.0> closing AMQP connection <0.1012.0> (10.0.0.48:46864 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:49.938 [info] <0.1054.0> accepting AMQP connection <0.1054.0> (10.0.0.48:46870 -> 10.0.0.48:5673) 2025-10-20 05:05:49.940 [info] <0.1054.0> connection <0.1054.0> (10.0.0.48:46870 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:50.334 [info] <0.896.0> closing AMQP connection <0.896.0> (10.0.0.48:46814 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:50.335 [info] <0.1076.0> accepting AMQP connection <0.1076.0> (10.0.0.48:46874 -> 10.0.0.48:5673) 2025-10-20 05:05:50.336 [info] <0.1076.0> connection <0.1076.0> (10.0.0.48:46874 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:50.393 [info] <0.878.0> closing AMQP connection <0.878.0> (10.0.0.49:56424 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:50.397 [info] <0.1098.0> accepting AMQP connection <0.1098.0> (10.0.0.49:56428 -> 10.0.0.48:5673) 2025-10-20 05:05:50.398 [info] <0.1098.0> connection <0.1098.0> (10.0.0.49:56428 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:50.808 [info] <0.932.0> closing AMQP connection <0.932.0> (10.0.0.254:40438 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:50.810 [info] <0.1120.0> accepting AMQP connection <0.1120.0> (10.0.0.254:40464 -> 10.0.0.48:5673) 2025-10-20 05:05:50.825 [info] <0.1120.0> connection <0.1120.0> (10.0.0.254:40464 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:50.953 [info] <0.1054.0> closing AMQP connection <0.1054.0> (10.0.0.48:46870 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:50.954 [info] <0.1144.0> accepting AMQP connection <0.1144.0> (10.0.0.48:46882 -> 10.0.0.48:5673) 2025-10-20 05:05:50.956 [info] <0.1144.0> connection <0.1144.0> (10.0.0.48:46882 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:51.977 [info] <0.1144.0> closing AMQP connection <0.1144.0> (10.0.0.48:46882 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:05:51.978 [info] <0.1187.0> accepting AMQP connection <0.1187.0> (10.0.0.48:53882 -> 10.0.0.48:5673) 2025-10-20 05:05:51.979 [info] <0.1187.0> connection <0.1187.0> (10.0.0.48:53882 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:05:55.901 [info] <0.1230.0> RabbitMQ is asked to stop... 2025-10-20 05:05:55.979 [info] <0.1230.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:05:55.980 [info] <0.1230.0> Stopping application 'rabbitmq_management' 2025-10-20 05:05:55.981 [warning] <0.566.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:05:55.982 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:05:55.982 [info] <0.1230.0> Stopping application 'amqp_client' 2025-10-20 05:05:55.983 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:05:55.983 [info] <0.1230.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:05:55.985 [info] <0.1230.0> Stopping application 'cowboy' 2025-10-20 05:05:55.985 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:05:55.985 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:05:55.986 [info] <0.1230.0> Stopping application 'cowlib' 2025-10-20 05:05:55.986 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:05:55.986 [info] <0.1230.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:05:55.990 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:05:55.990 [info] <0.1230.0> Stopping application 'rabbit' 2025-10-20 05:05:55.991 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:05:55.991 [info] <0.558.0> stopped TCP listener on [::]:5673 2025-10-20 05:05:55.992 [error] <0.1076.0> Error on AMQP connection <0.1076.0> (10.0.0.48:46874 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:05:55.992 [error] <0.1187.0> Error on AMQP connection <0.1187.0> (10.0.0.48:53882 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:05:55.992 [error] <0.1120.0> Error on AMQP connection <0.1120.0> (10.0.0.254:40464 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:05:55.992 [error] <0.951.0> Error on AMQP connection <0.951.0> (10.0.0.254:40450 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:05:55.992 [error] <0.1098.0> Error on AMQP connection <0.1098.0> (10.0.0.49:56428 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:05:55.992 [error] <0.1005.0> Error on AMQP connection <0.1005.0> (10.0.0.48:46854 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:05:55.992 [error] <0.860.0> Error on AMQP connection <0.860.0> (10.0.0.49:56408 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:05:55.992 [error] <0.914.0> Error on AMQP connection <0.914.0> (10.0.0.48:46828 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:05:55.999 [info] <0.482.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:05:55.999 [warning] <0.869.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:55.999 [warning] <0.923.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:55.999 [warning] <0.1032.0> Mirrored queue 'job_request_consumer' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:55.999 [warning] <0.1041.0> Mirrored queue 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:56.000 [warning] <0.1174.0> Mirrored queue 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-4371-2..tftp_queue' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:56.000 [warning] <0.1209.0> Mirrored queue 'device_manager_ztp.ztp_queue' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:56.000 [warning] <0.1131.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:56.000 [warning] <0.1165.0> Mirrored queue 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-4371-2..config_queue' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:56.000 [warning] <0.960.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:56.000 [warning] <0.1087.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:56.000 [warning] <0.1109.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:05:56.002 [info] <0.512.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' 2025-10-20 05:05:56.007 [info] <0.512.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped 2025-10-20 05:05:56.008 [info] <0.509.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' 2025-10-20 05:05:56.013 [info] <0.509.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped 2025-10-20 05:05:56.021 [info] <0.43.0> Application rabbit exited with reason: stopped 2025-10-20 05:05:56.021 [info] <0.1230.0> Stopping application 'mnesia' 2025-10-20 05:05:56.024 [info] <0.1230.0> Stopping application 'rabbit_common' 2025-10-20 05:05:56.024 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:05:56.024 [info] <0.43.0> Application rabbit_common exited with reason: stopped 2025-10-20 05:05:56.024 [info] <0.1230.0> Stopping application 'sysmon_handler' 2025-10-20 05:05:56.025 [info] <0.1230.0> Stopping application 'os_mon' 2025-10-20 05:05:56.025 [info] <0.43.0> Application sysmon_handler exited with reason: stopped 2025-10-20 05:05:56.027 [info] <0.1230.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:05:56.027 [info] <0.43.0> Application os_mon exited with reason: stopped 2025-10-20 05:06:07.742 [info] <0.1283.0> Resetting Rabbit forcefully 2025-10-20 05:06:09.443 [info] <0.1295.0> RabbitMQ is asked to stop... 2025-10-20 05:06:09.494 [info] <0.1295.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_management_agent rabbitmq_web_dispatch rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:06:09.494 [info] <0.1295.0> Stopping application 'rabbitmq_management' 2025-10-20 05:06:09.494 [info] <0.1295.0> Stopping application 'amqp_client' 2025-10-20 05:06:09.494 [info] <0.1295.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:06:09.494 [info] <0.1295.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:06:09.494 [info] <0.1295.0> Stopping application 'rabbit' 2025-10-20 05:06:09.494 [info] <0.1295.0> Stopping application 'mnesia' 2025-10-20 05:06:09.494 [info] <0.1295.0> Stopping application 'rabbit_common' 2025-10-20 05:06:09.494 [info] <0.1295.0> Stopping application 'sysmon_handler' 2025-10-20 05:06:09.494 [info] <0.1295.0> Stopping application 'os_mon' 2025-10-20 05:06:09.494 [info] <0.1295.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:06:09.494 [info] <0.1295.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 observer_cli tools credentials_obfuscation crypto stdout_formatter recon inets jsx xmerl lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:06:14.713 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:06:16.274 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:06:16.274 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:06:16.292 [info] <0.240.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:06:16.292 [info] <0.240.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:06:16.304 [info] <0.240.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:06:16.304 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:06:16.305 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:06:16.311 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:06:16.316 [info] <0.248.0> Enabling free disk space monitoring 2025-10-20 05:06:16.316 [info] <0.248.0> Disk free limit set to 50MB 2025-10-20 05:06:16.321 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:06:16.321 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:06:16.322 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:06:16.322 [info] <0.252.0> FHC read buffering: OFF 2025-10-20 05:06:16.322 [info] <0.252.0> FHC write buffering: ON 2025-10-20 05:06:16.323 [info] <0.240.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:06:16.323 [info] <0.241.0> Will use 4 processes for default worker pool 2025-10-20 05:06:16.323 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:06:16.323 [info] <0.240.0> Running boot step database defined by app rabbit 2025-10-20 05:06:16.323 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:06:16.323 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:06:16.323 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:06:16.323 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:06:16.324 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:06:16.324 [info] <0.240.0> Will wait for 37930 milliseconds before proceeding with registration... 2025-10-20 05:06:54.255 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:06:54.255 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:06:54.294 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:06:54.327 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:06:54.328 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:06:54.837 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:06:54.838 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:06:54.838 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:06:55.342 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:06:55.343 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:06:55.343 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:06:55.864 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:06:55.865 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:06:55.865 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:06:56.367 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:06:56.370 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:06:56.370 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:06:56.877 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:06:56.878 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:06:56.878 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:06:57.380 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:06:57.381 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:06:57.381 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:06:57.883 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:06:57.884 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:06:57.884 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:06:58.386 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:06:58.386 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:06:58.387 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:06:58.900 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:06:58.901 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:06:58.901 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:06:59.402 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:06:59.403 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:06:59.526 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:06:59.526 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:06:59.562 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:06:59.563 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:06:59.599 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:06:59.599 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:06:59.600 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:06:59.600 [info] <0.240.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:06:59.600 [info] <0.240.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:06:59.601 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:06:59.601 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:06:59.601 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:06:59.602 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:06:59.602 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:06:59.602 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:06:59.603 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:06:59.603 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:06:59.603 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:06:59.604 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:06:59.604 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:06:59.604 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:06:59.605 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:06:59.605 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:06:59.605 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:06:59.606 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:06:59.606 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:06:59.606 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:06:59.606 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:06:59.607 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:06:59.607 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:06:59.607 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:06:59.608 [info] <0.459.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:06:59.609 [info] <0.240.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:06:59.615 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:06:59.615 [info] <0.463.0> Starting rabbit_node_monitor 2025-10-20 05:06:59.615 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:06:59.615 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:06:59.615 [info] <0.240.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:06:59.615 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:06:59.646 [info] <0.240.0> message_store upgrades: 1 to apply 2025-10-20 05:06:59.646 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:06:59.647 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:06:59.647 [info] <0.240.0> message_store upgrades: Removing the old message store data 2025-10-20 05:06:59.650 [info] <0.240.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:06:59.676 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:06:59.676 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:06:59.676 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:06:59.677 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:06:59.677 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:06:59.677 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:06:59.677 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:06:59.677 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:06:59.678 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:06:59.678 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:06:59.678 [warning] <0.240.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:06:59.678 [info] <0.240.0> Management plugin: using rates mode 'basic' 2025-10-20 05:06:59.678 [info] <0.240.0> Running boot step recovery defined by app rabbit 2025-10-20 05:06:59.678 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:06:59.679 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:06:59.679 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:06:59.679 [info] <0.240.0> Importing users... 2025-10-20 05:06:59.679 [info] <0.240.0> Creating user 'guest' 2025-10-20 05:06:59.686 [info] <0.240.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:06:59.696 [info] <0.240.0> Importing vhosts... 2025-10-20 05:06:59.696 [info] <0.240.0> Adding vhost '/' 2025-10-20 05:06:59.724 [info] <0.511.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:06:59.729 [info] <0.511.0> Starting message stores for vhost '/' 2025-10-20 05:06:59.730 [info] <0.515.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:06:59.731 [info] <0.511.0> Started message store of type transient for vhost '/' 2025-10-20 05:06:59.731 [info] <0.518.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:06:59.732 [warning] <0.518.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:06:59.733 [info] <0.511.0> Started message store of type persistent for vhost '/' 2025-10-20 05:06:59.734 [info] <0.240.0> Importing user permissions... 2025-10-20 05:06:59.734 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:06:59.737 [info] <0.240.0> Importing topic permissions... 2025-10-20 05:06:59.737 [info] <0.240.0> Importing parameters... 2025-10-20 05:06:59.737 [info] <0.240.0> Importing global parameters... 2025-10-20 05:06:59.737 [info] <0.240.0> Importing policies... 2025-10-20 05:06:59.745 [info] <0.240.0> Importing queues... 2025-10-20 05:06:59.745 [info] <0.240.0> Importing exchanges... 2025-10-20 05:06:59.745 [info] <0.240.0> Importing bindings... 2025-10-20 05:06:59.745 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:06:59.745 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:06:59.745 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:06:59.745 [info] <0.240.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:06:59.745 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:06:59.749 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:06:59.753 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:06:59.753 [info] <0.240.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:06:59.753 [info] <0.240.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:06:59.753 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:06:59.753 [info] <0.240.0> Running boot step networking defined by app rabbit 2025-10-20 05:06:59.755 [info] <0.564.0> started TCP listener on [::]:5673 2025-10-20 05:06:59.755 [info] <0.240.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:06:59.755 [info] <0.240.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:06:59.756 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:06:59.756 [info] <0.566.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:06:59.793 [info] <0.616.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:06:59.793 [info] <0.722.0> Statistics database started. 2025-10-20 05:06:59.793 [info] <0.721.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:06:59.793 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:06:59.976 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch 2025-10-20 05:07:00.332 [info] <0.732.0> accepting AMQP connection <0.732.0> (10.0.0.254:45054 -> 10.0.0.48:5673) 2025-10-20 05:07:00.339 [info] <0.732.0> connection <0.732.0> (10.0.0.254:45054 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:07:00.402 [info] <0.752.0> accepting AMQP connection <0.752.0> (10.0.0.48:55246 -> 10.0.0.48:5673) 2025-10-20 05:07:00.405 [info] <0.752.0> connection <0.752.0> (10.0.0.48:55246 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:07:00.460 [info] <0.770.0> accepting AMQP connection <0.770.0> (10.0.0.49:57346 -> 10.0.0.48:5673) 2025-10-20 05:07:00.464 [info] <0.770.0> connection <0.770.0> (10.0.0.49:57346 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:07:01.010 [info] <0.789.0> accepting AMQP connection <0.789.0> (10.0.0.49:57358 -> 10.0.0.48:5673) 2025-10-20 05:07:01.011 [info] <0.792.0> accepting AMQP connection <0.792.0> (10.0.0.49:57374 -> 10.0.0.48:5673) 2025-10-20 05:07:01.013 [info] <0.792.0> connection <0.792.0> (10.0.0.49:57374 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:07:01.013 [info] <0.797.0> accepting AMQP connection <0.797.0> (10.0.0.48:55256 -> 10.0.0.48:5673) 2025-10-20 05:07:01.014 [info] <0.789.0> connection <0.789.0> (10.0.0.49:57358 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:07:01.014 [info] <0.805.0> accepting AMQP connection <0.805.0> (10.0.0.48:55262 -> 10.0.0.48:5673) 2025-10-20 05:07:01.016 [info] <0.797.0> connection <0.797.0> (10.0.0.48:55256 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:07:01.016 [info] <0.805.0> connection <0.805.0> (10.0.0.48:55262 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:07:01.016 [info] <0.826.0> accepting AMQP connection <0.826.0> (10.0.0.254:45068 -> 10.0.0.48:5673) 2025-10-20 05:07:01.019 [info] <0.826.0> connection <0.826.0> (10.0.0.254:45068 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:07:01.045 [info] <0.879.0> accepting AMQP connection <0.879.0> (10.0.0.254:45084 -> 10.0.0.48:5673) 2025-10-20 05:07:01.050 [info] <0.879.0> connection <0.879.0> (10.0.0.254:45084 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:07:01.655 [info] <0.898.0> RabbitMQ is asked to stop... 2025-10-20 05:07:01.760 [info] <0.898.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:07:01.760 [info] <0.898.0> Stopping application 'rabbitmq_management' 2025-10-20 05:07:01.763 [warning] <0.572.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:07:01.764 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:07:01.764 [info] <0.898.0> Stopping application 'amqp_client' 2025-10-20 05:07:01.767 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:07:01.767 [info] <0.898.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:07:01.770 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:07:01.770 [info] <0.898.0> Stopping application 'cowboy' 2025-10-20 05:07:01.773 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:07:01.773 [info] <0.898.0> Stopping application 'cowlib' 2025-10-20 05:07:01.773 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:07:01.773 [info] <0.898.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:07:01.778 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:07:01.778 [info] <0.898.0> Stopping application 'rabbit' 2025-10-20 05:07:01.778 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:07:01.778 [info] <0.564.0> stopped TCP listener on [::]:5673 2025-10-20 05:07:01.778 [error] <0.770.0> Error on AMQP connection <0.770.0> (10.0.0.49:57346 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:07:01.778 [error] <0.732.0> Error on AMQP connection <0.732.0> (10.0.0.254:45054 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:07:01.778 [error] <0.792.0> Error on AMQP connection <0.792.0> (10.0.0.49:57374 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:07:01.778 [error] <0.797.0> Error on AMQP connection <0.797.0> (10.0.0.48:55256 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:07:01.779 [error] <0.789.0> Error on AMQP connection <0.789.0> (10.0.0.49:57358 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:07:01.779 [error] <0.805.0> Error on AMQP connection <0.805.0> (10.0.0.48:55262 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:07:01.779 [error] <0.752.0> Error on AMQP connection <0.752.0> (10.0.0.48:55246 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:07:01.779 [error] <0.879.0> Error on AMQP connection <0.879.0> (10.0.0.254:45084 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:07:01.779 [error] <0.826.0> Error on AMQP connection <0.826.0> (10.0.0.254:45068 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:07:01.783 [info] <0.488.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:07:01.784 [warning] <0.831.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:07:01.784 [warning] <0.743.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:07:01.784 [warning] <0.835.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:07:01.784 [warning] <0.761.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:07:01.785 [warning] <0.888.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:07:01.785 [warning] <0.870.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:07:01.785 [warning] <0.779.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:07:01.785 [warning] <0.827.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:07:01.785 [warning] <0.838.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:07:01.786 [info] <0.518.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' 2025-10-20 05:07:02.824 [warning] lager_file_backend dropped 13 messages in the last second that exceeded the limit of 50 messages/sec 2025-10-20 05:07:12.598 [info] <0.949.0> Resetting Rabbit forcefully 2025-10-20 05:07:14.982 [info] <0.961.0> RabbitMQ is asked to stop... 2025-10-20 05:07:15.075 [info] <0.961.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_management_agent rabbitmq_web_dispatch rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:07:15.075 [info] <0.961.0> Stopping application 'rabbitmq_management' 2025-10-20 05:07:15.075 [info] <0.961.0> Stopping application 'amqp_client' 2025-10-20 05:07:15.075 [info] <0.961.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:07:15.075 [info] <0.961.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:07:15.075 [info] <0.961.0> Stopping application 'rabbit' 2025-10-20 05:07:15.075 [info] <0.961.0> Stopping application 'mnesia' 2025-10-20 05:07:15.075 [info] <0.961.0> Stopping application 'rabbit_common' 2025-10-20 05:07:15.075 [info] <0.961.0> Stopping application 'sysmon_handler' 2025-10-20 05:07:15.075 [info] <0.961.0> Stopping application 'os_mon' 2025-10-20 05:07:15.075 [info] <0.961.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:07:15.075 [info] <0.961.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 observer_cli tools credentials_obfuscation crypto stdout_formatter recon inets jsx xmerl lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:07:24.325 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:07:29.431 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:07:29.432 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:07:29.460 [info] <0.240.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:07:29.461 [info] <0.240.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:07:29.477 [info] <0.240.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:07:29.477 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:07:29.478 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:07:29.489 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:07:29.500 [info] <0.248.0> Enabling free disk space monitoring 2025-10-20 05:07:29.500 [info] <0.248.0> Disk free limit set to 50MB 2025-10-20 05:07:29.505 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:07:29.506 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:07:29.506 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:07:29.507 [info] <0.252.0> FHC read buffering: OFF 2025-10-20 05:07:29.507 [info] <0.252.0> FHC write buffering: ON 2025-10-20 05:07:29.508 [info] <0.240.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:07:29.509 [info] <0.241.0> Will use 4 processes for default worker pool 2025-10-20 05:07:29.509 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:07:29.510 [info] <0.240.0> Running boot step database defined by app rabbit 2025-10-20 05:07:29.510 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:07:29.511 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:07:29.511 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:07:29.511 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:07:29.512 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:07:29.512 [info] <0.240.0> Will wait for 58216 milliseconds before proceeding with registration... 2025-10-20 05:08:15.886 [info] <0.294.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping... 2025-10-20 05:08:27.728 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:08:27.729 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:08:27.732 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:08:27.736 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:08:27.736 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:08:28.238 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:08:28.239 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:08:28.240 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:08:28.741 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:08:28.742 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:08:28.742 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:08:29.245 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:08:29.246 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:08:29.246 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:08:29.747 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:08:29.748 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:08:29.748 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:08:30.250 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:08:30.251 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:08:30.252 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:08:30.753 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:08:30.754 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:08:30.754 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:08:31.255 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:08:31.256 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:08:31.256 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:08:31.757 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:08:31.758 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:08:31.758 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:08:32.259 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:08:32.260 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:08:32.260 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:08:32.760 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:08:32.763 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:08:32.873 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:08:32.873 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:08:32.901 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:08:32.901 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:08:32.930 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:08:32.931 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:08:32.931 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:08:32.931 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:08:32.932 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:08:32.932 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:08:32.932 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:08:32.932 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:08:32.932 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:08:32.932 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:08:32.932 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:08:32.932 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:08:32.932 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:08:32.932 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:08:32.932 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:08:32.935 [info] <0.463.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:08:32.935 [info] <0.240.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:08:32.940 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:08:32.940 [info] <0.467.0> Starting rabbit_node_monitor 2025-10-20 05:08:32.940 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:08:32.940 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:08:32.941 [info] <0.240.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:08:32.941 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:08:32.962 [info] <0.240.0> message_store upgrades: 1 to apply 2025-10-20 05:08:32.962 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:08:32.962 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:08:32.962 [info] <0.240.0> message_store upgrades: Removing the old message store data 2025-10-20 05:08:32.962 [info] <0.240.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:08:32.987 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:08:32.987 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:08:32.987 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:08:32.987 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:08:32.988 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:08:32.988 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:08:32.988 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:08:32.988 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:08:32.988 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:08:32.988 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:08:32.988 [warning] <0.240.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:08:32.989 [info] <0.240.0> Management plugin: using rates mode 'basic' 2025-10-20 05:08:32.989 [info] <0.240.0> Running boot step recovery defined by app rabbit 2025-10-20 05:08:32.989 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:08:32.990 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:08:32.990 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:08:32.990 [info] <0.240.0> Importing users... 2025-10-20 05:08:32.990 [info] <0.240.0> Creating user 'guest' 2025-10-20 05:08:32.999 [info] <0.240.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:08:33.001 [info] <0.240.0> Importing vhosts... 2025-10-20 05:08:33.002 [info] <0.240.0> Adding vhost '/' 2025-10-20 05:08:33.029 [info] <0.515.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:08:33.037 [info] <0.515.0> Starting message stores for vhost '/' 2025-10-20 05:08:33.038 [info] <0.519.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:08:33.039 [info] <0.515.0> Started message store of type transient for vhost '/' 2025-10-20 05:08:33.040 [info] <0.522.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:08:33.040 [warning] <0.522.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:08:33.041 [info] <0.515.0> Started message store of type persistent for vhost '/' 2025-10-20 05:08:33.042 [info] <0.240.0> Importing user permissions... 2025-10-20 05:08:33.042 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:08:33.048 [info] <0.240.0> Importing topic permissions... 2025-10-20 05:08:33.048 [info] <0.240.0> Importing parameters... 2025-10-20 05:08:33.048 [info] <0.240.0> Importing global parameters... 2025-10-20 05:08:33.048 [info] <0.240.0> Importing policies... 2025-10-20 05:08:33.056 [info] <0.240.0> Importing queues... 2025-10-20 05:08:33.056 [info] <0.240.0> Importing exchanges... 2025-10-20 05:08:33.056 [info] <0.240.0> Importing bindings... 2025-10-20 05:08:33.056 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:08:33.056 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:08:33.056 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:08:33.056 [info] <0.240.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:08:33.056 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:08:33.060 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:08:33.065 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:08:33.065 [info] <0.240.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:08:33.065 [info] <0.240.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:08:33.065 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:08:33.065 [info] <0.240.0> Running boot step networking defined by app rabbit 2025-10-20 05:08:33.071 [info] <0.568.0> started TCP listener on [::]:5673 2025-10-20 05:08:33.071 [info] <0.240.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:08:33.071 [info] <0.240.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:08:33.071 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:08:33.071 [info] <0.570.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:08:33.100 [info] <0.620.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:08:33.101 [info] <0.726.0> Statistics database started. 2025-10-20 05:08:33.101 [info] <0.725.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:08:33.101 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:08:33.340 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch 2025-10-20 05:08:33.365 [info] <0.294.0> RabbitMQ is asked to stop... 2025-10-20 05:08:33.416 [info] <0.294.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:08:33.416 [info] <0.294.0> Stopping application 'rabbitmq_management' 2025-10-20 05:08:33.418 [warning] <0.576.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:08:33.420 [info] <0.294.0> Stopping application 'amqp_client' 2025-10-20 05:08:33.420 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:08:33.421 [info] <0.294.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:08:33.422 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:08:33.423 [info] <0.294.0> Stopping application 'cowboy' 2025-10-20 05:08:33.423 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:08:33.424 [info] <0.294.0> Stopping application 'cowlib' 2025-10-20 05:08:33.424 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:08:33.424 [info] <0.294.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:08:33.424 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:08:33.426 [info] <0.294.0> Stopping application 'rabbit' 2025-10-20 05:08:33.426 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:08:33.426 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:08:33.426 [info] <0.568.0> stopped TCP listener on [::]:5673 2025-10-20 05:08:33.427 [info] <0.492.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:08:33.427 [info] <0.522.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' 2025-10-20 05:08:33.433 [info] <0.522.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped 2025-10-20 05:08:33.433 [info] <0.519.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' 2025-10-20 05:08:33.438 [info] <0.519.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped 2025-10-20 05:08:33.442 [info] <0.294.0> Stopping application 'mnesia' 2025-10-20 05:08:33.442 [info] <0.43.0> Application rabbit exited with reason: stopped 2025-10-20 05:08:33.448 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:08:33.448 [info] <0.294.0> Stopping application 'rabbit_common' 2025-10-20 05:08:33.448 [info] <0.43.0> Application rabbit_common exited with reason: stopped 2025-10-20 05:08:33.448 [info] <0.294.0> Stopping application 'sysmon_handler' 2025-10-20 05:08:33.449 [info] <0.43.0> Application sysmon_handler exited with reason: stopped 2025-10-20 05:08:33.449 [info] <0.294.0> Stopping application 'os_mon' 2025-10-20 05:08:33.450 [info] <0.43.0> Application os_mon exited with reason: stopped 2025-10-20 05:08:33.450 [info] <0.294.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:08:41.984 [info] <0.748.0> Resetting Rabbit forcefully 2025-10-20 05:08:44.060 [info] <0.760.0> RabbitMQ is asked to stop... 2025-10-20 05:08:44.179 [info] <0.760.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_management_agent rabbitmq_web_dispatch rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:08:44.179 [info] <0.760.0> Stopping application 'rabbitmq_management' 2025-10-20 05:08:44.179 [info] <0.760.0> Stopping application 'amqp_client' 2025-10-20 05:08:44.179 [info] <0.760.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:08:44.179 [info] <0.760.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:08:44.179 [info] <0.760.0> Stopping application 'rabbit' 2025-10-20 05:08:44.179 [info] <0.760.0> Stopping application 'mnesia' 2025-10-20 05:08:44.179 [info] <0.760.0> Stopping application 'rabbit_common' 2025-10-20 05:08:44.179 [info] <0.760.0> Stopping application 'sysmon_handler' 2025-10-20 05:08:44.179 [info] <0.760.0> Stopping application 'os_mon' 2025-10-20 05:08:44.179 [info] <0.760.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:08:44.179 [info] <0.760.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 observer_cli tools credentials_obfuscation crypto stdout_formatter recon inets jsx xmerl lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:08:53.567 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:08:56.350 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:08:56.351 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:08:56.376 [info] <0.240.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:08:56.378 [info] <0.240.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:08:56.390 [info] <0.240.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:08:56.390 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:08:56.390 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:08:56.405 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:08:56.412 [info] <0.248.0> Enabling free disk space monitoring 2025-10-20 05:08:56.412 [info] <0.248.0> Disk free limit set to 50MB 2025-10-20 05:08:56.416 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:08:56.416 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:08:56.416 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:08:56.416 [info] <0.252.0> FHC read buffering: OFF 2025-10-20 05:08:56.416 [info] <0.252.0> FHC write buffering: ON 2025-10-20 05:08:56.417 [info] <0.240.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:08:56.417 [info] <0.241.0> Will use 4 processes for default worker pool 2025-10-20 05:08:56.417 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:08:56.417 [info] <0.240.0> Running boot step database defined by app rabbit 2025-10-20 05:08:56.417 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:08:56.417 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:08:56.417 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:08:56.417 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:08:56.417 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:08:56.417 [info] <0.240.0> Will wait for 19336 milliseconds before proceeding with registration... 2025-10-20 05:09:15.754 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:09:15.754 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:09:15.762 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:09:15.768 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:09:15.768 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:09:16.271 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:09:16.272 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:09:16.272 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:09:16.774 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:09:16.775 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:09:16.776 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:09:17.279 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:09:17.280 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:09:17.280 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:09:17.782 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:09:17.783 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:09:17.783 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:09:18.297 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:09:18.306 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:09:18.306 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:09:18.807 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:09:18.808 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:09:18.808 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:09:19.311 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:09:19.317 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:09:19.317 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:09:19.818 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:09:19.819 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:09:19.819 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:09:20.321 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,tables_not_present} 2025-10-20 05:09:20.321 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,tables_not_present} 2025-10-20 05:09:20.322 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:09:20.822 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:09:20.824 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:09:20.931 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:09:20.931 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:09:20.961 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:09:20.961 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:09:20.993 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:09:20.993 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:09:20.994 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:09:20.994 [info] <0.240.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:09:20.995 [info] <0.240.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:09:20.995 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:09:20.995 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:09:20.996 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:09:20.996 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:09:20.996 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:09:20.997 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:09:20.997 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:09:20.998 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:09:20.998 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:09:20.998 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:09:20.999 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:09:20.999 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:09:21.000 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:09:21.000 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:09:21.000 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:09:21.001 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:09:21.001 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:09:21.001 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:09:21.002 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:09:21.002 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:09:21.002 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:09:21.003 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:09:21.004 [info] <0.447.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:09:21.004 [info] <0.240.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:09:21.010 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:09:21.011 [info] <0.451.0> Starting rabbit_node_monitor 2025-10-20 05:09:21.011 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:09:21.011 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:09:21.011 [info] <0.240.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:09:21.011 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:09:21.035 [info] <0.240.0> message_store upgrades: 1 to apply 2025-10-20 05:09:21.035 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:09:21.035 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:09:21.035 [info] <0.240.0> message_store upgrades: Removing the old message store data 2025-10-20 05:09:21.035 [info] <0.240.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:09:21.061 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:09:21.062 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:09:21.062 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:09:21.062 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:09:21.062 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:09:21.063 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:09:21.063 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:09:21.063 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:09:21.063 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:09:21.063 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:09:21.063 [warning] <0.240.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:09:21.063 [info] <0.240.0> Management plugin: using rates mode 'basic' 2025-10-20 05:09:21.063 [info] <0.240.0> Running boot step recovery defined by app rabbit 2025-10-20 05:09:21.064 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:09:21.064 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:09:21.064 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:09:21.064 [info] <0.240.0> Importing users... 2025-10-20 05:09:21.065 [info] <0.240.0> Creating user 'guest' 2025-10-20 05:09:21.070 [info] <0.240.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:09:21.074 [info] <0.240.0> Importing vhosts... 2025-10-20 05:09:21.074 [info] <0.240.0> Adding vhost '/' 2025-10-20 05:09:21.098 [info] <0.499.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:09:21.103 [info] <0.499.0> Starting message stores for vhost '/' 2025-10-20 05:09:21.103 [info] <0.503.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:09:21.104 [info] <0.499.0> Started message store of type transient for vhost '/' 2025-10-20 05:09:21.104 [info] <0.506.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:09:21.104 [warning] <0.506.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:09:21.105 [info] <0.499.0> Started message store of type persistent for vhost '/' 2025-10-20 05:09:21.106 [info] <0.240.0> Importing user permissions... 2025-10-20 05:09:21.106 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:09:21.109 [info] <0.240.0> Importing topic permissions... 2025-10-20 05:09:21.109 [info] <0.240.0> Importing parameters... 2025-10-20 05:09:21.109 [info] <0.240.0> Importing global parameters... 2025-10-20 05:09:21.109 [info] <0.240.0> Importing policies... 2025-10-20 05:09:21.115 [info] <0.240.0> Importing queues... 2025-10-20 05:09:21.115 [info] <0.240.0> Importing exchanges... 2025-10-20 05:09:21.115 [info] <0.240.0> Importing bindings... 2025-10-20 05:09:21.115 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:09:21.115 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:09:21.115 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:09:21.115 [info] <0.240.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:09:21.115 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:09:21.119 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:09:21.123 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:09:21.123 [info] <0.240.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:09:21.123 [info] <0.240.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:09:21.123 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:09:21.123 [info] <0.240.0> Running boot step networking defined by app rabbit 2025-10-20 05:09:21.126 [info] <0.552.0> started TCP listener on [::]:5673 2025-10-20 05:09:21.126 [info] <0.240.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:09:21.126 [info] <0.240.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:09:21.126 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:09:21.126 [info] <0.554.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:09:21.158 [info] <0.604.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:09:21.158 [info] <0.710.0> Statistics database started. 2025-10-20 05:09:21.158 [info] <0.709.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:09:21.158 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:09:21.259 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch 2025-10-20 05:09:21.815 [info] <0.720.0> accepting AMQP connection <0.720.0> (10.0.0.254:40850 -> 10.0.0.48:5673) 2025-10-20 05:09:21.820 [info] <0.724.0> accepting AMQP connection <0.724.0> (10.0.0.254:40852 -> 10.0.0.48:5673) 2025-10-20 05:09:21.820 [info] <0.727.0> accepting AMQP connection <0.727.0> (10.0.0.254:40864 -> 10.0.0.48:5673) 2025-10-20 05:09:21.821 [info] <0.731.0> accepting AMQP connection <0.731.0> (10.0.0.49:34344 -> 10.0.0.48:5673) 2025-10-20 05:09:21.821 [info] <0.720.0> connection <0.720.0> (10.0.0.254:40850 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:21.824 [info] <0.724.0> connection <0.724.0> (10.0.0.254:40852 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:21.824 [info] <0.731.0> connection <0.731.0> (10.0.0.49:34344 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:21.824 [info] <0.727.0> connection <0.727.0> (10.0.0.254:40864 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:21.825 [info] <0.755.0> accepting AMQP connection <0.755.0> (10.0.0.49:34368 -> 10.0.0.48:5673) 2025-10-20 05:09:21.825 [info] <0.745.0> accepting AMQP connection <0.745.0> (10.0.0.49:34358 -> 10.0.0.48:5673) 2025-10-20 05:09:21.828 [info] <0.745.0> connection <0.745.0> (10.0.0.49:34358 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:21.828 [info] <0.755.0> connection <0.755.0> (10.0.0.49:34368 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:21.834 [info] <0.823.0> accepting AMQP connection <0.823.0> (10.0.0.48:57934 -> 10.0.0.48:5673) 2025-10-20 05:09:21.834 [info] <0.826.0> accepting AMQP connection <0.826.0> (10.0.0.48:57948 -> 10.0.0.48:5673) 2025-10-20 05:09:21.837 [info] <0.835.0> accepting AMQP connection <0.835.0> (10.0.0.48:57950 -> 10.0.0.48:5673) 2025-10-20 05:09:21.839 [info] <0.826.0> connection <0.826.0> (10.0.0.48:57948 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:21.842 [info] <0.823.0> connection <0.823.0> (10.0.0.48:57934 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:21.857 [info] <0.835.0> connection <0.835.0> (10.0.0.48:57950 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:23.875 [info] <0.885.0> accepting AMQP connection <0.885.0> (10.0.0.48:57960 -> 10.0.0.48:5673) 2025-10-20 05:09:23.877 [info] <0.885.0> connection <0.885.0> (10.0.0.48:57960 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:23.895 [info] <0.903.0> accepting AMQP connection <0.903.0> (10.0.0.48:57974 -> 10.0.0.48:5673) 2025-10-20 05:09:23.899 [info] <0.903.0> connection <0.903.0> (10.0.0.48:57974 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:23.909 [info] <0.885.0> closing AMQP connection <0.885.0> (10.0.0.48:57960 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:09:23.909 [info] <0.903.0> closing AMQP connection <0.903.0> (10.0.0.48:57974 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:09:23.992 [info] <0.918.0> accepting AMQP connection <0.918.0> (10.0.0.254:58258 -> 10.0.0.48:5673) 2025-10-20 05:09:23.995 [info] <0.918.0> connection <0.918.0> (10.0.0.254:58258 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:24.030 [info] <0.936.0> accepting AMQP connection <0.936.0> (10.0.0.254:58260 -> 10.0.0.48:5673) 2025-10-20 05:09:24.035 [info] <0.936.0> connection <0.936.0> (10.0.0.254:58260 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:24.047 [info] <0.918.0> closing AMQP connection <0.918.0> (10.0.0.254:58258 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:09:24.047 [info] <0.936.0> closing AMQP connection <0.936.0> (10.0.0.254:58260 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:09:26.352 [info] <0.951.0> accepting AMQP connection <0.951.0> (10.0.0.48:57984 -> 10.0.0.48:5673) 2025-10-20 05:09:26.353 [info] <0.951.0> connection <0.951.0> (10.0.0.48:57984 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:28.701 [info] <0.1019.0> accepting AMQP connection <0.1019.0> (10.0.0.48:57998 -> 10.0.0.48:5673) 2025-10-20 05:09:28.703 [info] <0.1019.0> connection <0.1019.0> (10.0.0.48:57998 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:28.755 [info] <0.1046.0> accepting AMQP connection <0.1046.0> (10.0.0.48:58006 -> 10.0.0.48:5673) 2025-10-20 05:09:28.758 [info] <0.1046.0> connection <0.1046.0> (10.0.0.48:58006 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:28.773 [info] <0.1046.0> closing AMQP connection <0.1046.0> (10.0.0.48:58006 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:09:28.775 [info] <0.1057.0> accepting AMQP connection <0.1057.0> (10.0.0.48:58018 -> 10.0.0.48:5673) 2025-10-20 05:09:28.783 [info] <0.1057.0> connection <0.1057.0> (10.0.0.48:58018 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:29.251 [info] <0.1066.0> accepting AMQP connection <0.1066.0> (10.0.0.49:34382 -> 10.0.0.48:5673) 2025-10-20 05:09:29.255 [info] <0.1066.0> connection <0.1066.0> (10.0.0.49:34382 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:29.288 [info] <0.1084.0> accepting AMQP connection <0.1084.0> (10.0.0.49:34392 -> 10.0.0.48:5673) 2025-10-20 05:09:29.291 [info] <0.1084.0> connection <0.1084.0> (10.0.0.49:34392 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:29.302 [info] <0.1084.0> closing AMQP connection <0.1084.0> (10.0.0.49:34392 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:09:29.307 [info] <0.1095.0> accepting AMQP connection <0.1095.0> (10.0.0.49:34394 -> 10.0.0.48:5673) 2025-10-20 05:09:29.313 [info] <0.1095.0> connection <0.1095.0> (10.0.0.49:34394 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:29.896 [info] <0.1104.0> accepting AMQP connection <0.1104.0> (10.0.0.48:58030 -> 10.0.0.48:5673) 2025-10-20 05:09:29.899 [info] <0.1104.0> connection <0.1104.0> (10.0.0.48:58030 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:29.925 [info] <0.1122.0> accepting AMQP connection <0.1122.0> (10.0.0.48:58046 -> 10.0.0.48:5673) 2025-10-20 05:09:29.928 [info] <0.1122.0> connection <0.1122.0> (10.0.0.48:58046 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:29.942 [info] <0.1122.0> closing AMQP connection <0.1122.0> (10.0.0.48:58046 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:09:29.945 [info] <0.1133.0> accepting AMQP connection <0.1133.0> (10.0.0.48:58062 -> 10.0.0.48:5673) 2025-10-20 05:09:29.948 [info] <0.1133.0> connection <0.1133.0> (10.0.0.48:58062 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:09:30.067 [info] <0.1143.0> RabbitMQ is asked to stop... 2025-10-20 05:09:30.192 [info] <0.1143.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:09:30.192 [info] <0.1143.0> Stopping application 'rabbitmq_management' 2025-10-20 05:09:30.193 [warning] <0.560.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:09:30.195 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:09:30.195 [info] <0.1143.0> Stopping application 'amqp_client' 2025-10-20 05:09:30.197 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:09:30.197 [info] <0.1143.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:09:30.199 [info] <0.1143.0> Stopping application 'cowboy' 2025-10-20 05:09:30.199 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:09:30.201 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:09:30.201 [info] <0.1143.0> Stopping application 'cowlib' 2025-10-20 05:09:30.201 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:09:30.201 [info] <0.1143.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:09:30.202 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:09:30.203 [info] <0.1143.0> Stopping application 'rabbit' 2025-10-20 05:09:30.203 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:09:30.203 [info] <0.552.0> stopped TCP listener on [::]:5673 2025-10-20 05:09:30.205 [error] <0.727.0> Error on AMQP connection <0.727.0> (10.0.0.254:40864 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.205 [error] <0.724.0> Error on AMQP connection <0.724.0> (10.0.0.254:40852 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.826.0> Error on AMQP connection <0.826.0> (10.0.0.48:57948 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.835.0> Error on AMQP connection <0.835.0> (10.0.0.48:57950 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.731.0> Error on AMQP connection <0.731.0> (10.0.0.49:34344 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.951.0> Error on AMQP connection <0.951.0> (10.0.0.48:57984 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.755.0> Error on AMQP connection <0.755.0> (10.0.0.49:34368 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.1057.0> Error on AMQP connection <0.1057.0> (10.0.0.48:58018 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.1133.0> Error on AMQP connection <0.1133.0> (10.0.0.48:58062 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.720.0> Error on AMQP connection <0.720.0> (10.0.0.254:40850 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.1104.0> Error on AMQP connection <0.1104.0> (10.0.0.48:58030 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.1019.0> Error on AMQP connection <0.1019.0> (10.0.0.48:57998 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.1066.0> Error on AMQP connection <0.1066.0> (10.0.0.49:34382 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.823.0> Error on AMQP connection <0.823.0> (10.0.0.48:57934 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.206 [error] <0.745.0> Error on AMQP connection <0.745.0> (10.0.0.49:34358 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.207 [error] <0.1095.0> Error on AMQP connection <0.1095.0> (10.0.0.49:34394 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:09:30.212 [error] <0.850.0> Supervisor {<0.850.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error 2025-10-20 05:09:30.217 [info] <0.476.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:09:30.218 [warning] <0.990.0> Mirrored queue 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-4371-2..config_queue' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.218 [warning] <0.980.0> Mirrored queue 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.218 [warning] <0.1008.0> Mirrored queue 'device_manager_ztp.ztp_queue' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.218 [warning] <0.1113.0> Mirrored queue 'svc_monitor.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.219 [warning] <0.971.0> Mirrored queue 'job_request_consumer' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.219 [warning] <0.999.0> Mirrored queue 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-4371-2..tftp_queue' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.219 [warning] <0.764.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.219 [warning] <0.810.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.220 [warning] <0.853.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.220 [warning] <0.802.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.221 [warning] <0.857.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.221 [warning] <0.806.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.221 [warning] <0.1075.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-4371-3.:contrail-alarm-gen:0.cn-jenkins-deploy-platform-ansible-os-4371-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:30.221 [warning] <0.1037.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-4371-2.:contrail-alarm-gen:0.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:09:31.238 [warning] lager_file_backend dropped 17 messages in the last second that exceeded the limit of 50 messages/sec 2025-10-20 05:09:41.082 [info] <0.1226.0> Resetting Rabbit forcefully 2025-10-20 05:09:43.192 [info] <0.1238.0> RabbitMQ is asked to stop... 2025-10-20 05:09:43.386 [info] <0.1238.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_management_agent rabbitmq_web_dispatch rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:09:43.386 [info] <0.1238.0> Stopping application 'rabbitmq_management' 2025-10-20 05:09:43.386 [info] <0.1238.0> Stopping application 'amqp_client' 2025-10-20 05:09:43.387 [info] <0.1238.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:09:43.387 [info] <0.1238.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:09:43.387 [info] <0.1238.0> Stopping application 'rabbit' 2025-10-20 05:09:43.387 [info] <0.1238.0> Stopping application 'mnesia' 2025-10-20 05:09:43.388 [info] <0.1238.0> Stopping application 'rabbit_common' 2025-10-20 05:09:43.388 [info] <0.1238.0> Stopping application 'sysmon_handler' 2025-10-20 05:09:43.388 [info] <0.1238.0> Stopping application 'os_mon' 2025-10-20 05:09:43.388 [info] <0.1238.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:09:43.388 [info] <0.1238.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 observer_cli tools credentials_obfuscation crypto stdout_formatter recon inets jsx xmerl lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:09:51.829 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:09:54.286 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:09:54.287 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:09:54.312 [info] <0.240.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:09:54.313 [info] <0.240.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:09:54.326 [info] <0.240.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:09:54.326 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:09:54.327 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:09:54.335 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:09:54.344 [info] <0.248.0> Enabling free disk space monitoring 2025-10-20 05:09:54.344 [info] <0.248.0> Disk free limit set to 50MB 2025-10-20 05:09:54.348 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:09:54.349 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:09:54.349 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:09:54.349 [info] <0.252.0> FHC read buffering: OFF 2025-10-20 05:09:54.349 [info] <0.252.0> FHC write buffering: ON 2025-10-20 05:09:54.351 [info] <0.240.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:09:54.352 [info] <0.241.0> Will use 4 processes for default worker pool 2025-10-20 05:09:54.352 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:09:54.352 [info] <0.240.0> Running boot step database defined by app rabbit 2025-10-20 05:09:54.353 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:09:54.354 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:09:54.354 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:09:54.354 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:09:54.354 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:09:54.355 [info] <0.240.0> Will wait for 27157 milliseconds before proceeding with registration... 2025-10-20 05:10:21.512 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:10:21.512 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:10:21.515 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:10:21.516 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:10:21.516 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:10:22.018 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:10:22.019 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:10:22.019 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:10:22.705 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:10:22.707 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:10:22.707 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:10:23.214 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:10:23.215 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:10:23.215 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:10:23.722 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:10:23.724 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:10:23.724 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:10:24.230 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:10:24.232 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:10:24.232 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:10:24.741 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:10:24.743 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:10:24.743 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:10:25.250 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:10:25.251 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:10:25.251 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:10:25.770 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:10:25.771 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:10:25.771 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:10:26.277 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:10:26.278 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:10:26.278 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:10:26.779 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:10:26.783 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:10:26.961 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:10:26.961 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:10:27.017 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:10:27.017 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:10:27.078 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:10:27.078 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:10:27.078 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:10:27.078 [info] <0.240.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:10:27.078 [info] <0.240.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:10:27.078 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:10:27.078 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:10:27.079 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:10:27.080 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:10:27.080 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:10:27.080 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:10:27.080 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:10:27.080 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:10:27.080 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:10:27.080 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:10:27.080 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:10:27.080 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:10:27.081 [info] <0.466.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:10:27.081 [info] <0.240.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:10:27.086 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:10:27.087 [info] <0.470.0> Starting rabbit_node_monitor 2025-10-20 05:10:27.087 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:10:27.088 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:10:27.088 [info] <0.240.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:10:27.088 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:10:27.114 [info] <0.240.0> message_store upgrades: 1 to apply 2025-10-20 05:10:27.114 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:10:27.114 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:10:27.115 [info] <0.240.0> message_store upgrades: Removing the old message store data 2025-10-20 05:10:27.117 [info] <0.240.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:10:27.145 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:10:27.146 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:10:27.146 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:10:27.146 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:10:27.147 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:10:27.148 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:10:27.148 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:10:27.148 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:10:27.149 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:10:27.149 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:10:27.149 [warning] <0.240.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:10:27.150 [info] <0.240.0> Management plugin: using rates mode 'basic' 2025-10-20 05:10:27.150 [info] <0.240.0> Running boot step recovery defined by app rabbit 2025-10-20 05:10:27.151 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:10:27.152 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:10:27.152 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:10:27.153 [info] <0.240.0> Importing users... 2025-10-20 05:10:27.153 [info] <0.240.0> Creating user 'guest' 2025-10-20 05:10:27.183 [info] <0.240.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:10:27.196 [info] <0.240.0> Importing vhosts... 2025-10-20 05:10:27.196 [info] <0.240.0> Adding vhost '/' 2025-10-20 05:10:27.218 [info] <0.518.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:10:27.229 [info] <0.518.0> Starting message stores for vhost '/' 2025-10-20 05:10:27.229 [info] <0.522.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:10:27.235 [info] <0.518.0> Started message store of type transient for vhost '/' 2025-10-20 05:10:27.235 [info] <0.525.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:10:27.240 [warning] <0.525.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:10:27.243 [info] <0.518.0> Started message store of type persistent for vhost '/' 2025-10-20 05:10:27.244 [info] <0.240.0> Importing user permissions... 2025-10-20 05:10:27.244 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:10:27.251 [info] <0.240.0> Importing topic permissions... 2025-10-20 05:10:27.251 [info] <0.240.0> Importing parameters... 2025-10-20 05:10:27.252 [info] <0.240.0> Importing global parameters... 2025-10-20 05:10:27.252 [info] <0.240.0> Importing policies... 2025-10-20 05:10:27.263 [info] <0.240.0> Importing queues... 2025-10-20 05:10:27.263 [info] <0.240.0> Importing exchanges... 2025-10-20 05:10:27.263 [info] <0.240.0> Importing bindings... 2025-10-20 05:10:27.263 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:10:27.263 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:10:27.264 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:10:27.264 [info] <0.240.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:10:27.265 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:10:27.270 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:10:27.275 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:10:27.275 [info] <0.240.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:10:27.275 [info] <0.240.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:10:27.275 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:10:27.275 [info] <0.240.0> Running boot step networking defined by app rabbit 2025-10-20 05:10:27.278 [info] <0.571.0> started TCP listener on [::]:5673 2025-10-20 05:10:27.279 [info] <0.240.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:10:27.279 [info] <0.240.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:10:27.280 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:10:27.280 [info] <0.573.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:10:27.311 [info] <0.623.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:10:27.312 [info] <0.729.0> Statistics database started. 2025-10-20 05:10:27.312 [info] <0.728.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:10:27.313 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:10:27.485 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_web_dispatch * rabbitmq_management_agent 2025-10-20 05:10:29.117 [info] <0.470.0> node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-1' down: connection_closed 2025-10-20 05:10:29.591 [info] <0.752.0> accepting AMQP connection <0.752.0> (10.0.0.48:50938 -> 10.0.0.48:5673) 2025-10-20 05:10:29.593 [info] <0.752.0> connection <0.752.0> (10.0.0.48:50938 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.626 [info] <0.771.0> accepting AMQP connection <0.771.0> (10.0.0.48:50940 -> 10.0.0.48:5673) 2025-10-20 05:10:29.632 [info] <0.771.0> connection <0.771.0> (10.0.0.48:50940 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.657 [info] <0.771.0> closing AMQP connection <0.771.0> (10.0.0.48:50940 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:10:29.667 [info] <0.782.0> accepting AMQP connection <0.782.0> (10.0.0.49:60012 -> 10.0.0.48:5673) 2025-10-20 05:10:29.667 [info] <0.785.0> accepting AMQP connection <0.785.0> (10.0.0.48:50944 -> 10.0.0.48:5673) 2025-10-20 05:10:29.673 [info] <0.788.0> accepting AMQP connection <0.788.0> (10.0.0.48:50958 -> 10.0.0.48:5673) 2025-10-20 05:10:29.674 [info] <0.791.0> accepting AMQP connection <0.791.0> (10.0.0.49:60014 -> 10.0.0.48:5673) 2025-10-20 05:10:29.675 [info] <0.782.0> connection <0.782.0> (10.0.0.49:60012 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.676 [info] <0.788.0> connection <0.788.0> (10.0.0.48:50958 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.684 [info] <0.791.0> connection <0.791.0> (10.0.0.49:60014 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.691 [info] <0.785.0> connection <0.785.0> (10.0.0.48:50944 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.704 [info] <0.836.0> accepting AMQP connection <0.836.0> (10.0.0.48:50968 -> 10.0.0.48:5673) 2025-10-20 05:10:29.713 [info] <0.836.0> connection <0.836.0> (10.0.0.48:50968 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.714 [info] <0.849.0> accepting AMQP connection <0.849.0> (10.0.0.49:60026 -> 10.0.0.48:5673) 2025-10-20 05:10:29.730 [info] <0.857.0> accepting AMQP connection <0.857.0> (10.0.0.49:60028 -> 10.0.0.48:5673) 2025-10-20 05:10:29.742 [info] <0.862.0> accepting AMQP connection <0.862.0> (10.0.0.48:50978 -> 10.0.0.48:5673) 2025-10-20 05:10:29.742 [info] <0.836.0> closing AMQP connection <0.836.0> (10.0.0.48:50968 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:10:29.743 [info] <0.849.0> connection <0.849.0> (10.0.0.49:60026 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.746 [info] <0.857.0> connection <0.857.0> (10.0.0.49:60028 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.755 [info] <0.862.0> connection <0.862.0> (10.0.0.48:50978 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.761 [info] <0.849.0> closing AMQP connection <0.849.0> (10.0.0.49:60026 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:10:29.762 [info] <0.886.0> accepting AMQP connection <0.886.0> (10.0.0.49:60036 -> 10.0.0.48:5673) 2025-10-20 05:10:29.769 [info] <0.857.0> closing AMQP connection <0.857.0> (10.0.0.49:60028 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:10:29.772 [info] <0.890.0> accepting AMQP connection <0.890.0> (10.0.0.49:60038 -> 10.0.0.48:5673) 2025-10-20 05:10:29.772 [info] <0.886.0> connection <0.886.0> (10.0.0.49:60036 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:29.825 [info] <0.890.0> connection <0.890.0> (10.0.0.49:60038 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.221 [info] <0.905.0> accepting AMQP connection <0.905.0> (10.0.0.49:60050 -> 10.0.0.48:5673) 2025-10-20 05:10:30.221 [info] <0.908.0> accepting AMQP connection <0.908.0> (10.0.0.49:60052 -> 10.0.0.48:5673) 2025-10-20 05:10:30.225 [info] <0.908.0> connection <0.908.0> (10.0.0.49:60052 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.225 [info] <0.905.0> connection <0.905.0> (10.0.0.49:60050 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.225 [info] <0.915.0> accepting AMQP connection <0.915.0> (10.0.0.254:37200 -> 10.0.0.48:5673) 2025-10-20 05:10:30.226 [info] <0.922.0> accepting AMQP connection <0.922.0> (10.0.0.254:37216 -> 10.0.0.48:5673) 2025-10-20 05:10:30.227 [info] <0.929.0> accepting AMQP connection <0.929.0> (10.0.0.49:60056 -> 10.0.0.48:5673) 2025-10-20 05:10:30.227 [info] <0.932.0> accepting AMQP connection <0.932.0> (10.0.0.48:51008 -> 10.0.0.48:5673) 2025-10-20 05:10:30.227 [info] <0.938.0> accepting AMQP connection <0.938.0> (10.0.0.48:51004 -> 10.0.0.48:5673) 2025-10-20 05:10:30.227 [info] <0.935.0> accepting AMQP connection <0.935.0> (10.0.0.48:50994 -> 10.0.0.48:5673) 2025-10-20 05:10:30.234 [info] <0.941.0> accepting AMQP connection <0.941.0> (10.0.0.254:37222 -> 10.0.0.48:5673) 2025-10-20 05:10:30.235 [info] <0.915.0> connection <0.915.0> (10.0.0.254:37200 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.235 [info] <0.922.0> connection <0.922.0> (10.0.0.254:37216 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.236 [info] <0.932.0> connection <0.932.0> (10.0.0.48:51008 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.238 [info] <0.929.0> connection <0.929.0> (10.0.0.49:60056 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.242 [info] <0.935.0> connection <0.935.0> (10.0.0.48:50994 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.242 [info] <0.941.0> connection <0.941.0> (10.0.0.254:37222 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.245 [info] <0.938.0> connection <0.938.0> (10.0.0.48:51004 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.254 [info] <0.1041.0> accepting AMQP connection <0.1041.0> (10.0.0.254:37232 -> 10.0.0.48:5673) 2025-10-20 05:10:30.279 [info] <0.1041.0> connection <0.1041.0> (10.0.0.254:37232 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.298 [info] <0.1085.0> accepting AMQP connection <0.1085.0> (10.0.0.254:37242 -> 10.0.0.48:5673) 2025-10-20 05:10:30.310 [info] <0.1085.0> connection <0.1085.0> (10.0.0.254:37242 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.316 [info] <0.1085.0> closing AMQP connection <0.1085.0> (10.0.0.254:37242 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:10:30.319 [info] <0.1096.0> accepting AMQP connection <0.1096.0> (10.0.0.254:37258 -> 10.0.0.48:5673) 2025-10-20 05:10:30.324 [info] <0.1096.0> connection <0.1096.0> (10.0.0.254:37258 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:10:30.874 [info] <0.1106.0> RabbitMQ is asked to stop... 2025-10-20 05:10:30.975 [info] <0.1106.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:10:30.975 [info] <0.1106.0> Stopping application 'rabbitmq_management' 2025-10-20 05:10:30.977 [warning] <0.619.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:10:30.979 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:10:30.980 [info] <0.1106.0> Stopping application 'amqp_client' 2025-10-20 05:10:30.982 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:10:30.982 [info] <0.1106.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:10:30.984 [info] <0.1106.0> Stopping application 'cowboy' 2025-10-20 05:10:30.984 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:10:30.985 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:10:30.985 [info] <0.1106.0> Stopping application 'cowlib' 2025-10-20 05:10:30.986 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:10:30.986 [info] <0.1106.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:10:30.988 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:10:30.988 [info] <0.1106.0> Stopping application 'rabbit' 2025-10-20 05:10:30.989 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:10:30.989 [info] <0.571.0> stopped TCP listener on [::]:5673 2025-10-20 05:10:30.990 [error] <0.785.0> Error on AMQP connection <0.785.0> (10.0.0.48:50944 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.990 [error] <0.791.0> Error on AMQP connection <0.791.0> (10.0.0.49:60014 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.990 [error] <0.788.0> Error on AMQP connection <0.788.0> (10.0.0.48:50958 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.990 [error] <0.908.0> Error on AMQP connection <0.908.0> (10.0.0.49:60052 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.990 [error] <0.752.0> Error on AMQP connection <0.752.0> (10.0.0.48:50938 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.990 [error] <0.890.0> Error on AMQP connection <0.890.0> (10.0.0.49:60038 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.990 [error] <0.782.0> Error on AMQP connection <0.782.0> (10.0.0.49:60012 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.922.0> Error on AMQP connection <0.922.0> (10.0.0.254:37216 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.905.0> Error on AMQP connection <0.905.0> (10.0.0.49:60050 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.915.0> Error on AMQP connection <0.915.0> (10.0.0.254:37200 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.938.0> Error on AMQP connection <0.938.0> (10.0.0.48:51004 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.935.0> Error on AMQP connection <0.935.0> (10.0.0.48:50994 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.862.0> Error on AMQP connection <0.862.0> (10.0.0.48:50978 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.886.0> Error on AMQP connection <0.886.0> (10.0.0.49:60036 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.932.0> Error on AMQP connection <0.932.0> (10.0.0.48:51008 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.929.0> Error on AMQP connection <0.929.0> (10.0.0.49:60056 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.1041.0> Error on AMQP connection <0.1041.0> (10.0.0.254:37232 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.941.0> Error on AMQP connection <0.941.0> (10.0.0.254:37222 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.991 [error] <0.1096.0> Error on AMQP connection <0.1096.0> (10.0.0.254:37258 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:10:30.992 [error] <0.805.0> Supervisor {<0.805.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error 2025-10-20 05:10:30.999 [error] <0.793.0> Supervisor {<0.793.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error 2025-10-20 05:10:31.002 [error] <0.980.0> Supervisor {<0.980.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error 2025-10-20 05:10:31.003 [error] <0.945.0> Supervisor {<0.945.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error 2025-10-20 05:10:31.009 [info] <0.495.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:10:31.010 [warning] <0.812.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-4371-2.:contrail-topology:0.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.012 [warning] <0.1008.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.012 [warning] <0.1023.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.1015.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.1005.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.1076.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-4371-1.:contrail-topology:0.cn-jenkins-deploy-platform-ansible-os-4371-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.963.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.1028.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.762.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-4371-2.:contrail-snmp-collector:0.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.970.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.974.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.827.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-4371-3.:contrail-snmp-collector:0.cn-jenkins-deploy-platform-ansible-os-4371-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.839.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-4371-3.:contrail-topology:0.cn-jenkins-deploy-platform-ansible-os-4371-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.013 [warning] <0.983.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:10:31.077 [info] <0.525.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' 2025-10-20 05:10:31.084 [info] <0.525.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped 2025-10-20 05:10:31.085 [info] <0.522.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' 2025-10-20 05:10:31.089 [info] <0.522.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped 2025-10-20 05:10:31.096 [info] <0.1106.0> Stopping application 'mnesia' 2025-10-20 05:10:31.096 [info] <0.43.0> Application rabbit exited with reason: stopped 2025-10-20 05:10:31.114 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:10:31.114 [info] <0.1106.0> Stopping application 'rabbit_common' 2025-10-20 05:10:31.114 [info] <0.1106.0> Stopping application 'sysmon_handler' 2025-10-20 05:10:31.114 [info] <0.43.0> Application rabbit_common exited with reason: stopped 2025-10-20 05:10:31.121 [info] <0.43.0> Application sysmon_handler exited with reason: stopped 2025-10-20 05:10:31.121 [info] <0.1106.0> Stopping application 'os_mon' 2025-10-20 05:10:31.123 [info] <0.43.0> Application os_mon exited with reason: stopped 2025-10-20 05:10:32.122 [warning] lager_file_backend dropped 1 messages in the last second that exceeded the limit of 50 messages/sec 2025-10-20 05:10:50.312 [info] <0.1187.0> Resetting Rabbit forcefully 2025-10-20 05:10:52.423 [info] <0.1201.0> RabbitMQ is asked to stop... 2025-10-20 05:10:52.529 [info] <0.1201.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:10:52.529 [info] <0.1201.0> Stopping application 'rabbitmq_management' 2025-10-20 05:10:52.529 [info] <0.1201.0> Stopping application 'amqp_client' 2025-10-20 05:10:52.530 [info] <0.1201.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:10:52.530 [info] <0.1201.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:10:52.530 [info] <0.1201.0> Stopping application 'rabbit' 2025-10-20 05:10:52.530 [info] <0.1201.0> Stopping application 'mnesia' 2025-10-20 05:10:52.530 [info] <0.1201.0> Stopping application 'rabbit_common' 2025-10-20 05:10:52.530 [info] <0.1201.0> Stopping application 'sysmon_handler' 2025-10-20 05:10:52.530 [info] <0.1201.0> Stopping application 'os_mon' 2025-10-20 05:10:52.530 [info] <0.1201.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:10:52.530 [info] <0.1201.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 credentials_obfuscation observer_cli jsx tools xmerl crypto stdout_formatter recon inets lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:11:01.156 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:11:03.514 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:11:03.514 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:11:03.535 [info] <0.240.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:11:03.536 [info] <0.240.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:11:03.549 [info] <0.240.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:11:03.549 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:11:03.549 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:11:03.555 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:11:03.560 [info] <0.248.0> Enabling free disk space monitoring 2025-10-20 05:11:03.560 [info] <0.248.0> Disk free limit set to 50MB 2025-10-20 05:11:03.566 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:11:03.566 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:11:03.567 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:11:03.567 [info] <0.252.0> FHC read buffering: OFF 2025-10-20 05:11:03.567 [info] <0.252.0> FHC write buffering: ON 2025-10-20 05:11:03.567 [info] <0.240.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:11:03.567 [info] <0.241.0> Will use 4 processes for default worker pool 2025-10-20 05:11:03.567 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:11:03.567 [info] <0.240.0> Running boot step database defined by app rabbit 2025-10-20 05:11:03.568 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:11:03.568 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:11:03.568 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:11:03.568 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:11:03.568 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:11:03.568 [info] <0.240.0> Will wait for 32103 milliseconds before proceeding with registration... 2025-10-20 05:11:35.672 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:11:35.672 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:11:35.673 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:11:35.674 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:11:35.674 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:11:36.177 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:11:36.178 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:11:36.178 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:11:36.681 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:11:36.682 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:11:36.682 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:11:37.185 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:11:37.186 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:11:37.187 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:11:37.689 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:11:37.691 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:11:37.691 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:11:38.193 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:11:38.195 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:11:38.195 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:11:38.820 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:11:38.821 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:11:38.821 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:11:39.325 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:11:39.326 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:11:39.326 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:11:39.831 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:11:39.832 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:11:39.832 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:11:40.338 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:11:40.338 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:11:40.338 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:11:40.839 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:11:40.842 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:11:41.008 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:11:41.009 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:11:41.056 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:11:41.056 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:11:41.088 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:11:41.089 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:11:41.089 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:11:41.089 [info] <0.240.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:11:41.090 [info] <0.240.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:11:41.091 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:11:41.091 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:11:41.092 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:11:41.092 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:11:41.093 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:11:41.093 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:11:41.094 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:11:41.094 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:11:41.094 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:11:41.095 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:11:41.098 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:11:41.099 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:11:41.099 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:11:41.099 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:11:41.100 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:11:41.100 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:11:41.100 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:11:41.101 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:11:41.101 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:11:41.102 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:11:41.102 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:11:41.102 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:11:41.104 [info] <0.479.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:11:41.104 [info] <0.240.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:11:41.109 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:11:41.110 [info] <0.483.0> Starting rabbit_node_monitor 2025-10-20 05:11:41.110 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:11:41.110 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:11:41.110 [info] <0.240.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:11:41.110 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:11:41.134 [info] <0.240.0> message_store upgrades: 1 to apply 2025-10-20 05:11:41.134 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:11:41.134 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:11:41.135 [info] <0.240.0> message_store upgrades: Removing the old message store data 2025-10-20 05:11:41.138 [info] <0.240.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:11:41.168 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:11:41.169 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:11:41.169 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:11:41.170 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:11:41.170 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:11:41.171 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:11:41.172 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:11:41.172 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:11:41.173 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:11:41.173 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:11:41.174 [warning] <0.240.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:11:41.174 [info] <0.240.0> Management plugin: using rates mode 'basic' 2025-10-20 05:11:41.175 [info] <0.240.0> Running boot step recovery defined by app rabbit 2025-10-20 05:11:41.176 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:11:41.176 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:11:41.176 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:11:41.176 [info] <0.240.0> Importing users... 2025-10-20 05:11:41.176 [info] <0.240.0> Creating user 'guest' 2025-10-20 05:11:41.184 [info] <0.240.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:11:41.188 [info] <0.240.0> Importing vhosts... 2025-10-20 05:11:41.188 [info] <0.240.0> Adding vhost '/' 2025-10-20 05:11:41.224 [info] <0.531.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:11:41.231 [info] <0.531.0> Starting message stores for vhost '/' 2025-10-20 05:11:41.231 [info] <0.535.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:11:41.235 [info] <0.531.0> Started message store of type transient for vhost '/' 2025-10-20 05:11:41.235 [info] <0.538.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:11:41.236 [warning] <0.538.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:11:41.236 [info] <0.531.0> Started message store of type persistent for vhost '/' 2025-10-20 05:11:41.238 [info] <0.240.0> Importing user permissions... 2025-10-20 05:11:41.238 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:11:41.241 [info] <0.240.0> Importing topic permissions... 2025-10-20 05:11:41.241 [info] <0.240.0> Importing parameters... 2025-10-20 05:11:41.241 [info] <0.240.0> Importing global parameters... 2025-10-20 05:11:41.241 [info] <0.240.0> Importing policies... 2025-10-20 05:11:41.247 [info] <0.240.0> Importing queues... 2025-10-20 05:11:41.247 [info] <0.240.0> Importing exchanges... 2025-10-20 05:11:41.247 [info] <0.240.0> Importing bindings... 2025-10-20 05:11:41.248 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:11:41.248 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:11:41.248 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:11:41.248 [info] <0.240.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:11:41.248 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:11:41.252 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:11:41.255 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:11:41.255 [info] <0.240.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:11:41.255 [info] <0.240.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:11:41.255 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:11:41.256 [info] <0.240.0> Running boot step networking defined by app rabbit 2025-10-20 05:11:41.261 [info] <0.584.0> started TCP listener on [::]:5673 2025-10-20 05:11:41.262 [info] <0.240.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:11:41.262 [info] <0.240.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:11:41.262 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:11:41.262 [info] <0.586.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:11:41.321 [info] <0.636.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:11:41.322 [info] <0.742.0> Statistics database started. 2025-10-20 05:11:41.322 [info] <0.741.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:11:41.324 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:11:41.398 [info] <0.752.0> accepting AMQP connection <0.752.0> (10.0.0.49:59874 -> 10.0.0.48:5673) 2025-10-20 05:11:41.400 [info] <0.756.0> accepting AMQP connection <0.756.0> (10.0.0.48:57154 -> 10.0.0.48:5673) 2025-10-20 05:11:41.401 [info] <0.762.0> accepting AMQP connection <0.762.0> (10.0.0.48:57156 -> 10.0.0.48:5673) 2025-10-20 05:11:41.401 [info] <0.759.0> accepting AMQP connection <0.759.0> (10.0.0.254:55602 -> 10.0.0.48:5673) 2025-10-20 05:11:41.403 [info] <0.762.0> connection <0.762.0> (10.0.0.48:57156 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.403 [info] <0.756.0> connection <0.756.0> (10.0.0.48:57154 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.404 [info] <0.782.0> accepting AMQP connection <0.782.0> (10.0.0.49:59888 -> 10.0.0.48:5673) 2025-10-20 05:11:41.405 [info] <0.785.0> accepting AMQP connection <0.785.0> (10.0.0.254:55608 -> 10.0.0.48:5673) 2025-10-20 05:11:41.406 [info] <0.759.0> connection <0.759.0> (10.0.0.254:55602 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.406 [info] <0.795.0> accepting AMQP connection <0.795.0> (10.0.0.49:59904 -> 10.0.0.48:5673) 2025-10-20 05:11:41.407 [info] <0.752.0> connection <0.752.0> (10.0.0.49:59874 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.408 [info] <0.782.0> connection <0.782.0> (10.0.0.49:59888 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.409 [info] <0.817.0> accepting AMQP connection <0.817.0> (10.0.0.48:57166 -> 10.0.0.48:5673) 2025-10-20 05:11:41.409 [info] <0.821.0> accepting AMQP connection <0.821.0> (10.0.0.254:55620 -> 10.0.0.48:5673) 2025-10-20 05:11:41.410 [info] <0.795.0> connection <0.795.0> (10.0.0.49:59904 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.410 [info] <0.785.0> connection <0.785.0> (10.0.0.254:55608 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.411 [info] <0.817.0> connection <0.817.0> (10.0.0.48:57166 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.412 [info] <0.821.0> connection <0.821.0> (10.0.0.254:55620 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.532 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch 2025-10-20 05:11:41.825 [info] <0.928.0> accepting AMQP connection <0.928.0> (10.0.0.49:59906 -> 10.0.0.48:5673) 2025-10-20 05:11:41.838 [info] <0.928.0> connection <0.928.0> (10.0.0.49:59906 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.860 [info] <0.946.0> accepting AMQP connection <0.946.0> (10.0.0.49:59910 -> 10.0.0.48:5673) 2025-10-20 05:11:41.865 [info] <0.946.0> connection <0.946.0> (10.0.0.49:59910 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:41.882 [info] <0.946.0> closing AMQP connection <0.946.0> (10.0.0.49:59910 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:11:41.882 [info] <0.957.0> accepting AMQP connection <0.957.0> (10.0.0.49:59918 -> 10.0.0.48:5673) 2025-10-20 05:11:41.899 [info] <0.957.0> connection <0.957.0> (10.0.0.49:59918 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:42.387 [info] <0.967.0> accepting AMQP connection <0.967.0> (10.0.0.48:50010 -> 10.0.0.48:5673) 2025-10-20 05:11:42.390 [info] <0.967.0> connection <0.967.0> (10.0.0.48:50010 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:42.395 [info] <0.981.0> accepting AMQP connection <0.981.0> (10.0.0.49:59930 -> 10.0.0.48:5673) 2025-10-20 05:11:42.402 [info] <0.981.0> connection <0.981.0> (10.0.0.49:59930 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:42.473 [info] <0.1003.0> accepting AMQP connection <0.1003.0> (10.0.0.48:50016 -> 10.0.0.48:5673) 2025-10-20 05:11:42.474 [info] <0.1006.0> accepting AMQP connection <0.1006.0> (10.0.0.48:50030 -> 10.0.0.48:5673) 2025-10-20 05:11:42.476 [info] <0.1003.0> connection <0.1003.0> (10.0.0.48:50016 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:42.477 [info] <0.1006.0> connection <0.1006.0> (10.0.0.48:50030 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:11:42.716 [info] <0.1088.0> RabbitMQ is asked to stop... 2025-10-20 05:11:42.789 [info] <0.1088.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:11:42.789 [info] <0.1088.0> Stopping application 'rabbitmq_management' 2025-10-20 05:11:42.791 [warning] <0.592.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:11:42.792 [info] <0.1088.0> Stopping application 'amqp_client' 2025-10-20 05:11:42.792 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:11:42.794 [info] <0.1088.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:11:42.794 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:11:42.796 [info] <0.1088.0> Stopping application 'cowboy' 2025-10-20 05:11:42.796 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:11:42.797 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:11:42.797 [info] <0.1088.0> Stopping application 'cowlib' 2025-10-20 05:11:42.797 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:11:42.797 [info] <0.1088.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:11:42.799 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:11:42.799 [info] <0.1088.0> Stopping application 'rabbit' 2025-10-20 05:11:42.799 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:11:42.799 [info] <0.584.0> stopped TCP listener on [::]:5673 2025-10-20 05:11:42.800 [error] <0.752.0> Error on AMQP connection <0.752.0> (10.0.0.49:59874 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.821.0> Error on AMQP connection <0.821.0> (10.0.0.254:55620 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.785.0> Error on AMQP connection <0.785.0> (10.0.0.254:55608 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.756.0> Error on AMQP connection <0.756.0> (10.0.0.48:57154 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.762.0> Error on AMQP connection <0.762.0> (10.0.0.48:57156 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.928.0> Error on AMQP connection <0.928.0> (10.0.0.49:59906 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.957.0> Error on AMQP connection <0.957.0> (10.0.0.49:59918 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.981.0> Error on AMQP connection <0.981.0> (10.0.0.49:59930 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.782.0> Error on AMQP connection <0.782.0> (10.0.0.49:59888 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.759.0> Error on AMQP connection <0.759.0> (10.0.0.254:55602 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.967.0> Error on AMQP connection <0.967.0> (10.0.0.48:50010 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.1003.0> Error on AMQP connection <0.1003.0> (10.0.0.48:50016 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.817.0> Error on AMQP connection <0.817.0> (10.0.0.48:57166 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.795.0> Error on AMQP connection <0.795.0> (10.0.0.49:59904 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.800 [error] <0.1006.0> Error on AMQP connection <0.1006.0> (10.0.0.48:50030 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:11:42.803 [error] <0.765.0> Supervisor {<0.765.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error 2025-10-20 05:11:42.806 [error] <0.831.0> Supervisor {<0.831.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason noproc in context shutdown_error 2025-10-20 05:11:42.827 [info] <0.508.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:11:42.828 [warning] <0.788.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:11:42.828 [warning] <0.851.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:11:42.828 [warning] <0.1032.0> Mirrored queue 'job_request_consumer' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:11:42.828 [warning] <0.994.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-4371-3.:contrail-alarm-gen:0.cn-jenkins-deploy-platform-ansible-os-4371-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:11:42.828 [warning] <0.857.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:11:42.829 [warning] <0.976.0> Mirrored queue 'svc_monitor.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:11:43.481 [warning] lager_file_backend dropped 26 messages in the last second that exceeded the limit of 50 messages/sec 2025-10-20 05:12:09.051 [info] <0.1174.0> Resetting Rabbit forcefully 2025-10-20 05:12:11.071 [info] <0.1194.0> RabbitMQ is asked to stop... 2025-10-20 05:12:11.180 [info] <0.1194.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_management_agent rabbitmq_web_dispatch rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:12:11.180 [info] <0.1194.0> Stopping application 'rabbitmq_management' 2025-10-20 05:12:11.180 [info] <0.1194.0> Stopping application 'amqp_client' 2025-10-20 05:12:11.180 [info] <0.1194.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:12:11.180 [info] <0.1194.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:12:11.180 [info] <0.1194.0> Stopping application 'rabbit' 2025-10-20 05:12:11.181 [info] <0.1194.0> Stopping application 'mnesia' 2025-10-20 05:12:11.181 [info] <0.1194.0> Stopping application 'rabbit_common' 2025-10-20 05:12:11.181 [info] <0.1194.0> Stopping application 'sysmon_handler' 2025-10-20 05:12:11.181 [info] <0.1194.0> Stopping application 'os_mon' 2025-10-20 05:12:11.181 [info] <0.1194.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:12:11.181 [info] <0.1194.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 observer_cli tools credentials_obfuscation crypto stdout_formatter recon inets jsx xmerl lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:12:17.911 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:12:20.134 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:12:20.134 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:12:20.155 [info] <0.253.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:12:20.156 [info] <0.253.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:12:20.168 [info] <0.253.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:12:20.168 [info] <0.253.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:12:20.169 [info] <0.253.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:12:20.173 [info] <0.259.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:12:20.180 [info] <0.261.0> Enabling free disk space monitoring 2025-10-20 05:12:20.180 [info] <0.261.0> Disk free limit set to 50MB 2025-10-20 05:12:20.186 [info] <0.253.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:12:20.186 [info] <0.253.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:12:20.187 [info] <0.264.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:12:20.187 [info] <0.265.0> FHC read buffering: OFF 2025-10-20 05:12:20.187 [info] <0.265.0> FHC write buffering: ON 2025-10-20 05:12:20.188 [info] <0.253.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:12:20.188 [info] <0.254.0> Will use 4 processes for default worker pool 2025-10-20 05:12:20.188 [info] <0.254.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:12:20.188 [info] <0.253.0> Running boot step database defined by app rabbit 2025-10-20 05:12:20.189 [info] <0.253.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:12:20.189 [info] <0.253.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:12:20.189 [info] <0.253.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:12:20.189 [info] <0.253.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:12:20.189 [info] <0.253.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:12:20.189 [info] <0.253.0> Will wait for 31550 milliseconds before proceeding with registration... 2025-10-20 05:12:51.740 [info] <0.253.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:12:51.740 [info] <0.253.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:12:51.744 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:12:51.745 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:12:51.745 [error] <0.253.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:12:52.247 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:12:52.248 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:12:52.248 [error] <0.253.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:12:52.750 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:12:52.751 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:12:52.751 [error] <0.253.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:12:53.253 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:12:53.255 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:12:53.255 [error] <0.253.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:12:53.757 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:12:53.758 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:12:53.758 [error] <0.253.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:12:54.304 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:12:54.305 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:12:54.306 [error] <0.253.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:12:54.809 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:12:54.810 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:12:54.810 [error] <0.253.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:12:55.312 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:12:55.316 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:12:55.316 [error] <0.253.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:12:55.819 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:12:55.942 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:12:55.942 [error] <0.253.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:12:56.448 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:12:56.471 [warning] <0.253.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:12:56.471 [error] <0.253.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:12:56.972 [warning] <0.253.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:12:56.973 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:12:57.119 [info] <0.253.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:12:57.119 [info] <0.253.0> Successfully synced tables from a peer 2025-10-20 05:12:57.146 [info] <0.253.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:12:57.146 [info] <0.253.0> Successfully synced tables from a peer 2025-10-20 05:12:57.173 [info] <0.253.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:12:57.173 [info] <0.253.0> Successfully synced tables from a peer 2025-10-20 05:12:57.173 [info] <0.253.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:12:57.174 [info] <0.253.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:12:57.175 [info] <0.253.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:12:57.175 [info] <0.253.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:12:57.175 [info] <0.253.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:12:57.175 [info] <0.253.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:12:57.175 [info] <0.253.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:12:57.175 [info] <0.253.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:12:57.175 [info] <0.253.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:12:57.175 [info] <0.253.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:12:57.175 [info] <0.253.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:12:57.175 [info] <0.253.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:12:57.176 [info] <0.500.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:12:57.176 [info] <0.253.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:12:57.180 [info] <0.253.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:12:57.180 [info] <0.504.0> Starting rabbit_node_monitor 2025-10-20 05:12:57.180 [info] <0.253.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:12:57.180 [info] <0.253.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:12:57.180 [info] <0.253.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:12:57.180 [info] <0.253.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:12:57.201 [info] <0.253.0> message_store upgrades: 1 to apply 2025-10-20 05:12:57.201 [info] <0.253.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:12:57.201 [info] <0.253.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:12:57.201 [info] <0.253.0> message_store upgrades: Removing the old message store data 2025-10-20 05:12:57.201 [info] <0.253.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:12:57.226 [info] <0.253.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:12:57.226 [info] <0.253.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:12:57.226 [info] <0.253.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:12:57.227 [info] <0.253.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:12:57.227 [info] <0.253.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:12:57.228 [info] <0.253.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:12:57.229 [info] <0.253.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:12:57.229 [info] <0.253.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:12:57.229 [info] <0.253.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:12:57.230 [info] <0.253.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:12:57.230 [warning] <0.253.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:12:57.231 [info] <0.253.0> Management plugin: using rates mode 'basic' 2025-10-20 05:12:57.231 [info] <0.253.0> Running boot step recovery defined by app rabbit 2025-10-20 05:12:57.232 [info] <0.253.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:12:57.234 [info] <0.253.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:12:57.234 [info] <0.253.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:12:57.234 [info] <0.253.0> Importing users... 2025-10-20 05:12:57.235 [info] <0.253.0> Creating user 'guest' 2025-10-20 05:12:57.242 [info] <0.253.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:12:57.246 [info] <0.253.0> Importing vhosts... 2025-10-20 05:12:57.246 [info] <0.253.0> Adding vhost '/' 2025-10-20 05:12:57.275 [info] <0.552.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:12:57.279 [info] <0.552.0> Starting message stores for vhost '/' 2025-10-20 05:12:57.279 [info] <0.556.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:12:57.280 [info] <0.552.0> Started message store of type transient for vhost '/' 2025-10-20 05:12:57.280 [info] <0.559.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:12:57.281 [warning] <0.559.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:12:57.281 [info] <0.552.0> Started message store of type persistent for vhost '/' 2025-10-20 05:12:57.282 [info] <0.253.0> Importing user permissions... 2025-10-20 05:12:57.282 [info] <0.253.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:12:57.286 [info] <0.253.0> Importing topic permissions... 2025-10-20 05:12:57.286 [info] <0.253.0> Importing parameters... 2025-10-20 05:12:57.286 [info] <0.253.0> Importing global parameters... 2025-10-20 05:12:57.286 [info] <0.253.0> Importing policies... 2025-10-20 05:12:57.293 [info] <0.253.0> Importing queues... 2025-10-20 05:12:57.293 [info] <0.253.0> Importing exchanges... 2025-10-20 05:12:57.293 [info] <0.253.0> Importing bindings... 2025-10-20 05:12:57.293 [info] <0.253.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:12:57.294 [info] <0.253.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:12:57.294 [info] <0.253.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:12:57.294 [info] <0.253.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:12:57.294 [info] <0.253.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:12:57.299 [info] <0.253.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:12:57.303 [info] <0.253.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:12:57.304 [info] <0.253.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:12:57.304 [info] <0.253.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:12:57.304 [info] <0.253.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:12:57.304 [info] <0.253.0> Running boot step networking defined by app rabbit 2025-10-20 05:12:57.306 [info] <0.605.0> started TCP listener on [::]:5673 2025-10-20 05:12:57.306 [info] <0.253.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:12:57.306 [info] <0.253.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:12:57.306 [info] <0.253.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:12:57.306 [info] <0.607.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:12:57.341 [info] <0.657.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:12:57.341 [info] <0.763.0> Statistics database started. 2025-10-20 05:12:57.341 [info] <0.762.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:12:57.342 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:12:57.521 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch 2025-10-20 05:12:57.535 [info] <0.773.0> accepting AMQP connection <0.773.0> (10.0.0.48:60420 -> 10.0.0.48:5673) 2025-10-20 05:12:57.542 [info] <0.773.0> connection <0.773.0> (10.0.0.48:60420 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:12:58.339 [info] <0.504.0> node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-1' down: connection_closed 2025-10-20 05:12:59.891 [info] <0.787.0> accepting AMQP connection <0.787.0> (10.0.0.48:60430 -> 10.0.0.48:5673) 2025-10-20 05:12:59.893 [info] <0.787.0> connection <0.787.0> (10.0.0.48:60430 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:12:59.911 [info] <0.805.0> accepting AMQP connection <0.805.0> (10.0.0.48:60438 -> 10.0.0.48:5673) 2025-10-20 05:12:59.912 [info] <0.805.0> connection <0.805.0> (10.0.0.48:60438 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:12:59.918 [info] <0.805.0> closing AMQP connection <0.805.0> (10.0.0.48:60438 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:12:59.920 [info] <0.816.0> accepting AMQP connection <0.816.0> (10.0.0.48:60452 -> 10.0.0.48:5673) 2025-10-20 05:12:59.927 [info] <0.816.0> connection <0.816.0> (10.0.0.48:60452 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:01.208 [info] <0.825.0> accepting AMQP connection <0.825.0> (10.0.0.254:34192 -> 10.0.0.48:5673) 2025-10-20 05:13:01.209 [info] <0.828.0> accepting AMQP connection <0.828.0> (10.0.0.49:40988 -> 10.0.0.48:5673) 2025-10-20 05:13:01.210 [info] <0.825.0> connection <0.825.0> (10.0.0.254:34192 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:01.210 [info] <0.828.0> connection <0.828.0> (10.0.0.49:40988 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:01.213 [info] <0.843.0> accepting AMQP connection <0.843.0> (10.0.0.49:40996 -> 10.0.0.48:5673) 2025-10-20 05:13:01.213 [info] <0.849.0> accepting AMQP connection <0.849.0> (10.0.0.48:60460 -> 10.0.0.48:5673) 2025-10-20 05:13:01.214 [info] <0.853.0> accepting AMQP connection <0.853.0> (10.0.0.48:60462 -> 10.0.0.48:5673) 2025-10-20 05:13:01.214 [info] <0.849.0> connection <0.849.0> (10.0.0.48:60460 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:01.214 [info] <0.843.0> connection <0.843.0> (10.0.0.49:40996 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:01.215 [info] <0.853.0> connection <0.853.0> (10.0.0.48:60462 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:01.216 [info] <0.885.0> accepting AMQP connection <0.885.0> (10.0.0.254:34194 -> 10.0.0.48:5673) 2025-10-20 05:13:01.218 [info] <0.905.0> accepting AMQP connection <0.905.0> (10.0.0.48:60472 -> 10.0.0.48:5673) 2025-10-20 05:13:01.218 [info] <0.885.0> connection <0.885.0> (10.0.0.254:34194 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:01.219 [info] <0.905.0> connection <0.905.0> (10.0.0.48:60472 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:01.223 [info] <0.940.0> accepting AMQP connection <0.940.0> (10.0.0.254:34200 -> 10.0.0.48:5673) 2025-10-20 05:13:01.226 [info] <0.954.0> accepting AMQP connection <0.954.0> (10.0.0.49:41004 -> 10.0.0.48:5673) 2025-10-20 05:13:01.227 [info] <0.940.0> connection <0.940.0> (10.0.0.254:34200 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:01.233 [info] <0.954.0> connection <0.954.0> (10.0.0.49:41004 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:03.502 [info] <0.504.0> node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-1' up 2025-10-20 05:13:04.238 [info] <0.1002.0> RabbitMQ is asked to stop... 2025-10-20 05:13:04.305 [info] <0.1002.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:13:04.305 [info] <0.1002.0> Stopping application 'rabbitmq_management' 2025-10-20 05:13:04.308 [warning] <0.613.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:13:04.309 [info] <0.1002.0> Stopping application 'amqp_client' 2025-10-20 05:13:04.309 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:13:04.313 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:13:04.313 [info] <0.1002.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:13:04.315 [info] <0.1002.0> Stopping application 'cowboy' 2025-10-20 05:13:04.315 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:13:04.316 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:13:04.316 [info] <0.1002.0> Stopping application 'cowlib' 2025-10-20 05:13:04.316 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:13:04.316 [info] <0.1002.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:13:04.318 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:13:04.318 [info] <0.1002.0> Stopping application 'rabbit' 2025-10-20 05:13:04.318 [info] <0.253.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:13:04.318 [info] <0.605.0> stopped TCP listener on [::]:5673 2025-10-20 05:13:04.319 [error] <0.843.0> Error on AMQP connection <0.843.0> (10.0.0.49:40996 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.319 [error] <0.773.0> Error on AMQP connection <0.773.0> (10.0.0.48:60420 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.319 [error] <0.954.0> Error on AMQP connection <0.954.0> (10.0.0.49:41004 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.319 [error] <0.787.0> Error on AMQP connection <0.787.0> (10.0.0.48:60430 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.319 [error] <0.816.0> Error on AMQP connection <0.816.0> (10.0.0.48:60452 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.319 [error] <0.853.0> Error on AMQP connection <0.853.0> (10.0.0.48:60462 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.319 [error] <0.828.0> Error on AMQP connection <0.828.0> (10.0.0.49:40988 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.319 [error] <0.825.0> Error on AMQP connection <0.825.0> (10.0.0.254:34192 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.319 [error] <0.940.0> Error on AMQP connection <0.940.0> (10.0.0.254:34200 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.319 [error] <0.905.0> Error on AMQP connection <0.905.0> (10.0.0.48:60472 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.320 [error] <0.885.0> Error on AMQP connection <0.885.0> (10.0.0.254:34194 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.320 [error] <0.849.0> Error on AMQP connection <0.849.0> (10.0.0.48:60460 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:04.321 [error] <0.818.0> Supervisor {<0.818.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error 2025-10-20 05:13:04.321 [error] <0.789.0> Supervisor {<0.789.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error 2025-10-20 05:13:04.336 [error] <0.956.0> Supervisor {<0.956.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason noproc in context shutdown_error 2025-10-20 05:13:04.347 [info] <0.529.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:13:04.349 [warning] <0.978.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:04.355 [warning] <0.858.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:04.355 [warning] <0.932.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:04.361 [warning] <0.889.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:04.361 [warning] <0.963.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:04.361 [warning] <0.848.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:04.361 [warning] <0.796.0> Mirrored queue 'device_manager.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:04.361 [warning] <0.892.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:04.361 [warning] <0.937.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:04.362 [warning] <0.898.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:04.365 [info] <0.559.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' 2025-10-20 05:13:04.369 [info] <0.559.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped 2025-10-20 05:13:04.369 [info] <0.556.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' 2025-10-20 05:13:04.375 [info] <0.556.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped 2025-10-20 05:13:04.421 [info] <0.43.0> Application rabbit exited with reason: stopped 2025-10-20 05:13:04.421 [info] <0.1002.0> Stopping application 'mnesia' 2025-10-20 05:13:05.192 [warning] lager_file_backend dropped 8 messages in the last second that exceeded the limit of 50 messages/sec 2025-10-20 05:13:23.245 [info] <0.1069.0> Resetting Rabbit forcefully 2025-10-20 05:13:25.401 [info] <0.1084.0> RabbitMQ is asked to stop... 2025-10-20 05:13:25.466 [info] <0.1084.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_management_agent rabbitmq_web_dispatch rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:13:25.466 [info] <0.1084.0> Stopping application 'rabbitmq_management' 2025-10-20 05:13:25.466 [info] <0.1084.0> Stopping application 'amqp_client' 2025-10-20 05:13:25.466 [info] <0.1084.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:13:25.466 [info] <0.1084.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:13:25.466 [info] <0.1084.0> Stopping application 'rabbit' 2025-10-20 05:13:25.466 [info] <0.1084.0> Stopping application 'mnesia' 2025-10-20 05:13:25.466 [info] <0.1084.0> Stopping application 'rabbit_common' 2025-10-20 05:13:25.466 [info] <0.1084.0> Stopping application 'sysmon_handler' 2025-10-20 05:13:25.466 [info] <0.1084.0> Stopping application 'os_mon' 2025-10-20 05:13:25.466 [info] <0.1084.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:13:25.466 [info] <0.1084.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 observer_cli tools credentials_obfuscation crypto stdout_formatter recon inets jsx xmerl lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:13:33.207 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:13:35.028 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:13:35.028 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:13:35.050 [info] <0.240.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:13:35.051 [info] <0.240.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:13:35.062 [info] <0.240.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:13:35.062 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:13:35.063 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:13:35.068 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:13:35.073 [info] <0.248.0> Enabling free disk space monitoring 2025-10-20 05:13:35.073 [info] <0.248.0> Disk free limit set to 50MB 2025-10-20 05:13:35.077 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:13:35.077 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:13:35.077 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:13:35.078 [info] <0.252.0> FHC read buffering: OFF 2025-10-20 05:13:35.078 [info] <0.252.0> FHC write buffering: ON 2025-10-20 05:13:35.078 [info] <0.240.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:13:35.078 [info] <0.241.0> Will use 4 processes for default worker pool 2025-10-20 05:13:35.078 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:13:35.078 [info] <0.240.0> Running boot step database defined by app rabbit 2025-10-20 05:13:35.078 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:13:35.078 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:13:35.078 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:13:35.078 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:13:35.078 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:13:35.079 [info] <0.240.0> Will wait for 11785 milliseconds before proceeding with registration... 2025-10-20 05:13:46.865 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:13:46.865 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:13:46.866 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:13:46.867 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:13:46.867 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:13:47.369 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:13:47.370 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:13:47.370 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:13:47.872 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:13:47.873 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:13:47.873 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:13:48.375 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:13:48.376 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:13:48.378 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:13:48.881 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:13:48.883 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:13:48.883 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:13:49.385 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:13:49.386 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:13:49.386 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:13:49.890 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:13:49.891 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:13:49.891 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:13:50.393 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:13:50.394 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:13:50.394 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:13:50.896 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:13:50.897 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:13:50.897 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:13:51.399 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:13:51.400 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:13:51.400 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:13:51.900 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:13:51.903 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:13:52.039 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:13:52.039 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:13:52.066 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:13:52.066 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:13:52.095 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:13:52.095 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:13:52.095 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:13:52.095 [info] <0.240.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:13:52.095 [info] <0.240.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:13:52.095 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:13:52.095 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:13:52.095 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:13:52.095 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:13:52.095 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:13:52.095 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:13:52.096 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:13:52.096 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:13:52.097 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:13:52.097 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:13:52.097 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:13:52.097 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:13:52.097 [info] <0.473.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:13:52.098 [info] <0.240.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:13:52.101 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:13:52.101 [info] <0.477.0> Starting rabbit_node_monitor 2025-10-20 05:13:52.101 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:13:52.102 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:13:52.102 [info] <0.240.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:13:52.102 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:13:52.124 [info] <0.240.0> message_store upgrades: 1 to apply 2025-10-20 05:13:52.125 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:13:52.125 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:13:52.125 [info] <0.240.0> message_store upgrades: Removing the old message store data 2025-10-20 05:13:52.125 [info] <0.240.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:13:52.149 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:13:52.149 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:13:52.149 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:13:52.150 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:13:52.150 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:13:52.151 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:13:52.151 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:13:52.151 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:13:52.151 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:13:52.151 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:13:52.151 [warning] <0.240.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:13:52.151 [info] <0.240.0> Management plugin: using rates mode 'basic' 2025-10-20 05:13:52.151 [info] <0.240.0> Running boot step recovery defined by app rabbit 2025-10-20 05:13:52.152 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:13:52.152 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:13:52.152 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:13:52.153 [info] <0.240.0> Importing users... 2025-10-20 05:13:52.153 [info] <0.240.0> Creating user 'guest' 2025-10-20 05:13:52.159 [info] <0.240.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:13:52.161 [info] <0.240.0> Importing vhosts... 2025-10-20 05:13:52.161 [info] <0.240.0> Adding vhost '/' 2025-10-20 05:13:52.181 [info] <0.525.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:13:52.186 [info] <0.525.0> Starting message stores for vhost '/' 2025-10-20 05:13:52.186 [info] <0.529.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:13:52.187 [info] <0.525.0> Started message store of type transient for vhost '/' 2025-10-20 05:13:52.187 [info] <0.532.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:13:52.187 [warning] <0.532.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:13:52.188 [info] <0.525.0> Started message store of type persistent for vhost '/' 2025-10-20 05:13:52.189 [info] <0.240.0> Importing user permissions... 2025-10-20 05:13:52.189 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:13:52.191 [info] <0.240.0> Importing topic permissions... 2025-10-20 05:13:52.191 [info] <0.240.0> Importing parameters... 2025-10-20 05:13:52.191 [info] <0.240.0> Importing global parameters... 2025-10-20 05:13:52.191 [info] <0.240.0> Importing policies... 2025-10-20 05:13:52.198 [info] <0.240.0> Importing queues... 2025-10-20 05:13:52.198 [info] <0.240.0> Importing exchanges... 2025-10-20 05:13:52.198 [info] <0.240.0> Importing bindings... 2025-10-20 05:13:52.198 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:13:52.198 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:13:52.198 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:13:52.198 [info] <0.240.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:13:52.198 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:13:52.203 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:13:52.209 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:13:52.209 [info] <0.240.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:13:52.209 [info] <0.240.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:13:52.209 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:13:52.209 [info] <0.240.0> Running boot step networking defined by app rabbit 2025-10-20 05:13:52.214 [info] <0.578.0> started TCP listener on [::]:5673 2025-10-20 05:13:52.214 [info] <0.240.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:13:52.214 [info] <0.240.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:13:52.214 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:13:52.214 [info] <0.580.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:13:52.240 [info] <0.630.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:13:52.240 [info] <0.736.0> Statistics database started. 2025-10-20 05:13:52.240 [info] <0.735.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:13:52.241 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:13:52.399 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch 2025-10-20 05:13:54.332 [info] <0.747.0> accepting AMQP connection <0.747.0> (10.0.0.49:39158 -> 10.0.0.48:5673) 2025-10-20 05:13:54.332 [info] <0.750.0> accepting AMQP connection <0.750.0> (10.0.0.49:39170 -> 10.0.0.48:5673) 2025-10-20 05:13:54.333 [info] <0.754.0> accepting AMQP connection <0.754.0> (10.0.0.48:47248 -> 10.0.0.48:5673) 2025-10-20 05:13:54.334 [info] <0.750.0> connection <0.750.0> (10.0.0.49:39170 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:54.337 [info] <0.754.0> connection <0.754.0> (10.0.0.48:47248 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:54.337 [info] <0.747.0> connection <0.747.0> (10.0.0.49:39158 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:54.343 [info] <0.803.0> accepting AMQP connection <0.803.0> (10.0.0.254:38204 -> 10.0.0.48:5673) 2025-10-20 05:13:54.344 [info] <0.806.0> accepting AMQP connection <0.806.0> (10.0.0.254:38220 -> 10.0.0.48:5673) 2025-10-20 05:13:54.344 [info] <0.809.0> accepting AMQP connection <0.809.0> (10.0.0.48:47250 -> 10.0.0.48:5673) 2025-10-20 05:13:54.346 [info] <0.803.0> connection <0.803.0> (10.0.0.254:38204 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:54.347 [info] <0.819.0> accepting AMQP connection <0.819.0> (10.0.0.254:38226 -> 10.0.0.48:5673) 2025-10-20 05:13:54.347 [info] <0.809.0> connection <0.809.0> (10.0.0.48:47250 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:54.348 [info] <0.806.0> connection <0.806.0> (10.0.0.254:38220 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:54.348 [info] <0.829.0> accepting AMQP connection <0.829.0> (10.0.0.48:47252 -> 10.0.0.48:5673) 2025-10-20 05:13:54.353 [info] <0.829.0> connection <0.829.0> (10.0.0.48:47252 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:54.353 [info] <0.819.0> connection <0.819.0> (10.0.0.254:38226 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:54.357 [info] <0.887.0> accepting AMQP connection <0.887.0> (10.0.0.49:39172 -> 10.0.0.48:5673) 2025-10-20 05:13:54.360 [info] <0.887.0> connection <0.887.0> (10.0.0.49:39172 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:13:57.540 [info] <0.923.0> RabbitMQ is asked to stop... 2025-10-20 05:13:57.634 [info] <0.923.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:13:57.634 [info] <0.923.0> Stopping application 'rabbitmq_management' 2025-10-20 05:13:57.636 [warning] <0.586.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:13:57.638 [info] <0.923.0> Stopping application 'amqp_client' 2025-10-20 05:13:57.638 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:13:57.639 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:13:57.639 [info] <0.923.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:13:57.640 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:13:57.640 [info] <0.923.0> Stopping application 'cowboy' 2025-10-20 05:13:57.642 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:13:57.642 [info] <0.923.0> Stopping application 'cowlib' 2025-10-20 05:13:57.642 [info] <0.923.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:13:57.642 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:13:57.643 [info] <0.923.0> Stopping application 'rabbit' 2025-10-20 05:13:57.643 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:13:57.643 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:13:57.643 [info] <0.578.0> stopped TCP listener on [::]:5673 2025-10-20 05:13:57.644 [error] <0.829.0> Error on AMQP connection <0.829.0> (10.0.0.48:47252 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:57.644 [error] <0.887.0> Error on AMQP connection <0.887.0> (10.0.0.49:39172 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:57.644 [error] <0.750.0> Error on AMQP connection <0.750.0> (10.0.0.49:39170 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:57.644 [error] <0.809.0> Error on AMQP connection <0.809.0> (10.0.0.48:47250 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:57.644 [error] <0.747.0> Error on AMQP connection <0.747.0> (10.0.0.49:39158 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:57.644 [error] <0.803.0> Error on AMQP connection <0.803.0> (10.0.0.254:38204 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:57.644 [error] <0.806.0> Error on AMQP connection <0.806.0> (10.0.0.254:38220 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:57.644 [error] <0.819.0> Error on AMQP connection <0.819.0> (10.0.0.254:38226 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:57.644 [error] <0.754.0> Error on AMQP connection <0.754.0> (10.0.0.48:47248 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:13:57.645 [error] <0.820.0> Supervisor {<0.820.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error 2025-10-20 05:13:57.646 [warning] <0.779.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:57.646 [warning] <0.784.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:57.647 [warning] <0.836.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:57.647 [warning] <0.776.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:57.647 [warning] <0.875.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:57.647 [warning] <0.848.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:57.647 [warning] <0.879.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:57.647 [warning] <0.840.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:57.648 [warning] <0.902.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:13:57.651 [info] <0.532.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' 2025-10-20 05:13:57.654 [info] <0.502.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:13:57.665 [info] <0.532.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped 2025-10-20 05:13:57.665 [info] <0.529.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' 2025-10-20 05:13:57.674 [info] <0.529.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped 2025-10-20 05:13:57.680 [info] <0.923.0> Stopping application 'mnesia' 2025-10-20 05:13:57.680 [info] <0.43.0> Application rabbit exited with reason: stopped 2025-10-20 05:13:57.684 [info] <0.923.0> Stopping application 'rabbit_common' 2025-10-20 05:13:57.684 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:13:57.685 [info] <0.43.0> Application rabbit_common exited with reason: stopped 2025-10-20 05:13:57.685 [info] <0.923.0> Stopping application 'sysmon_handler' 2025-10-20 05:13:57.686 [info] <0.43.0> Application sysmon_handler exited with reason: stopped 2025-10-20 05:13:57.686 [info] <0.923.0> Stopping application 'os_mon' 2025-10-20 05:13:58.149 [warning] lager_file_backend dropped 2 messages in the last second that exceeded the limit of 50 messages/sec 2025-10-20 05:14:05.442 [info] <0.974.0> Resetting Rabbit forcefully 2025-10-20 05:14:07.438 [info] <0.988.0> RabbitMQ is asked to stop... 2025-10-20 05:14:07.508 [info] <0.988.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_management_agent rabbitmq_web_dispatch rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:14:07.508 [info] <0.988.0> Stopping application 'rabbitmq_management' 2025-10-20 05:14:07.508 [info] <0.988.0> Stopping application 'amqp_client' 2025-10-20 05:14:07.509 [info] <0.988.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:14:07.509 [info] <0.988.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:14:07.509 [info] <0.988.0> Stopping application 'rabbit' 2025-10-20 05:14:07.509 [info] <0.988.0> Stopping application 'mnesia' 2025-10-20 05:14:07.509 [info] <0.988.0> Stopping application 'rabbit_common' 2025-10-20 05:14:07.509 [info] <0.988.0> Stopping application 'sysmon_handler' 2025-10-20 05:14:07.509 [info] <0.988.0> Stopping application 'os_mon' 2025-10-20 05:14:07.509 [info] <0.988.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:14:07.509 [info] <0.988.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 observer_cli tools credentials_obfuscation crypto stdout_formatter recon inets jsx xmerl lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:14:13.301 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:14:15.866 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:14:15.866 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:14:15.894 [info] <0.246.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:14:15.895 [info] <0.246.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:14:15.910 [info] <0.246.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:14:15.911 [info] <0.246.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:14:15.911 [info] <0.246.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:14:15.927 [info] <0.252.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:14:15.934 [info] <0.254.0> Enabling free disk space monitoring 2025-10-20 05:14:15.934 [info] <0.254.0> Disk free limit set to 50MB 2025-10-20 05:14:15.941 [info] <0.246.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:14:15.941 [info] <0.246.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:14:15.942 [info] <0.257.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:14:15.942 [info] <0.258.0> FHC read buffering: OFF 2025-10-20 05:14:15.942 [info] <0.258.0> FHC write buffering: ON 2025-10-20 05:14:15.943 [info] <0.246.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:14:15.943 [info] <0.247.0> Will use 4 processes for default worker pool 2025-10-20 05:14:15.943 [info] <0.247.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:14:15.944 [info] <0.246.0> Running boot step database defined by app rabbit 2025-10-20 05:14:15.945 [info] <0.246.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:14:15.945 [info] <0.246.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:14:15.945 [info] <0.246.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:14:15.945 [info] <0.246.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:14:15.945 [info] <0.246.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:14:15.945 [info] <0.246.0> Will wait for 32010 milliseconds before proceeding with registration... 2025-10-20 05:14:47.955 [info] <0.246.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:14:47.955 [info] <0.246.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:14:47.957 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:14:47.958 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:14:47.958 [error] <0.246.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:14:48.459 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:14:48.461 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:14:48.461 [error] <0.246.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:14:48.964 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:14:48.966 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {badrpc,nodedown} 2025-10-20 05:14:48.966 [error] <0.246.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:14:49.467 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:14:49.519 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:14:49.519 [error] <0.246.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:14:50.022 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:14:50.023 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:14:50.023 [error] <0.246.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:14:50.524 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:14:50.526 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:14:50.526 [error] <0.246.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:14:51.028 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:14:51.038 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:14:51.038 [error] <0.246.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:14:51.545 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:14:51.551 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:14:51.552 [error] <0.246.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:14:52.053 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:14:52.057 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:14:52.057 [error] <0.246.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:14:52.560 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:14:52.567 [warning] <0.246.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:14:52.567 [error] <0.246.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:14:53.067 [warning] <0.246.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:14:53.069 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:14:53.181 [info] <0.246.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:14:53.181 [info] <0.246.0> Successfully synced tables from a peer 2025-10-20 05:14:53.214 [info] <0.246.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:14:53.215 [info] <0.246.0> Successfully synced tables from a peer 2025-10-20 05:14:53.252 [info] <0.246.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:14:53.252 [info] <0.246.0> Successfully synced tables from a peer 2025-10-20 05:14:53.253 [info] <0.246.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:14:53.253 [info] <0.246.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:14:53.253 [info] <0.246.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:14:53.254 [info] <0.246.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:14:53.254 [info] <0.246.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:14:53.255 [info] <0.246.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:14:53.255 [info] <0.246.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:14:53.255 [info] <0.246.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:14:53.256 [info] <0.246.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:14:53.256 [info] <0.246.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:14:53.257 [info] <0.246.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:14:53.257 [info] <0.246.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:14:53.257 [info] <0.246.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:14:53.258 [info] <0.246.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:14:53.258 [info] <0.246.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:14:53.258 [info] <0.246.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:14:53.259 [info] <0.246.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:14:53.259 [info] <0.246.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:14:53.260 [info] <0.246.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:14:53.260 [info] <0.246.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:14:53.260 [info] <0.246.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:14:53.261 [info] <0.246.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:14:53.261 [info] <0.246.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:14:53.261 [info] <0.246.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:14:53.262 [info] <0.246.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:14:53.263 [info] <0.489.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:14:53.263 [info] <0.246.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:14:53.268 [info] <0.246.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:14:53.269 [info] <0.493.0> Starting rabbit_node_monitor 2025-10-20 05:14:53.269 [info] <0.246.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:14:53.269 [info] <0.246.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:14:53.270 [info] <0.246.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:14:53.270 [info] <0.246.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:14:53.295 [info] <0.246.0> message_store upgrades: 1 to apply 2025-10-20 05:14:53.295 [info] <0.246.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:14:53.296 [info] <0.246.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:14:53.296 [info] <0.246.0> message_store upgrades: Removing the old message store data 2025-10-20 05:14:53.299 [info] <0.246.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:14:53.333 [info] <0.246.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:14:53.333 [info] <0.246.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:14:53.333 [info] <0.246.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:14:53.334 [info] <0.246.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:14:53.335 [info] <0.246.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:14:53.336 [info] <0.246.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:14:53.336 [info] <0.246.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:14:53.337 [info] <0.246.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:14:53.337 [info] <0.246.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:14:53.337 [info] <0.246.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:14:53.338 [warning] <0.246.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:14:53.338 [info] <0.246.0> Management plugin: using rates mode 'basic' 2025-10-20 05:14:53.339 [info] <0.246.0> Running boot step recovery defined by app rabbit 2025-10-20 05:14:53.340 [info] <0.246.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:14:53.341 [info] <0.246.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:14:53.341 [info] <0.246.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:14:53.342 [info] <0.246.0> Importing users... 2025-10-20 05:14:53.342 [info] <0.246.0> Creating user 'guest' 2025-10-20 05:14:53.349 [info] <0.246.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:14:53.352 [info] <0.246.0> Importing vhosts... 2025-10-20 05:14:53.353 [info] <0.246.0> Adding vhost '/' 2025-10-20 05:14:53.375 [info] <0.541.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:14:53.381 [info] <0.541.0> Starting message stores for vhost '/' 2025-10-20 05:14:53.381 [info] <0.545.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:14:53.382 [info] <0.541.0> Started message store of type transient for vhost '/' 2025-10-20 05:14:53.383 [info] <0.548.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:14:53.384 [warning] <0.548.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:14:53.385 [info] <0.541.0> Started message store of type persistent for vhost '/' 2025-10-20 05:14:53.386 [info] <0.246.0> Importing user permissions... 2025-10-20 05:14:53.386 [info] <0.246.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:14:53.391 [info] <0.246.0> Importing topic permissions... 2025-10-20 05:14:53.391 [info] <0.246.0> Importing parameters... 2025-10-20 05:14:53.391 [info] <0.246.0> Importing global parameters... 2025-10-20 05:14:53.392 [info] <0.246.0> Importing policies... 2025-10-20 05:14:53.400 [info] <0.246.0> Importing queues... 2025-10-20 05:14:53.400 [info] <0.246.0> Importing exchanges... 2025-10-20 05:14:53.401 [info] <0.246.0> Importing bindings... 2025-10-20 05:14:53.401 [info] <0.246.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:14:53.401 [info] <0.246.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:14:53.402 [info] <0.246.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:14:53.402 [info] <0.246.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:14:53.403 [info] <0.246.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:14:53.415 [info] <0.246.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:14:53.420 [info] <0.246.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:14:53.421 [info] <0.246.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:14:53.421 [info] <0.246.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:14:53.421 [info] <0.246.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:14:53.421 [info] <0.246.0> Running boot step networking defined by app rabbit 2025-10-20 05:14:53.426 [info] <0.594.0> started TCP listener on [::]:5673 2025-10-20 05:14:53.427 [info] <0.246.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:14:53.427 [info] <0.246.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:14:53.427 [info] <0.246.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:14:53.427 [info] <0.596.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:14:53.466 [info] <0.646.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:14:53.467 [info] <0.752.0> Statistics database started. 2025-10-20 05:14:53.467 [info] <0.751.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:14:53.467 [notice] <0.110.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:14:53.614 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch 2025-10-20 05:14:54.461 [info] <0.762.0> accepting AMQP connection <0.762.0> (10.0.0.48:35174 -> 10.0.0.48:5673) 2025-10-20 05:14:54.467 [info] <0.762.0> connection <0.762.0> (10.0.0.48:35174 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:55.045 [info] <0.783.0> accepting AMQP connection <0.783.0> (10.0.0.254:49354 -> 10.0.0.48:5673) 2025-10-20 05:14:55.050 [info] <0.783.0> connection <0.783.0> (10.0.0.254:49354 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:55.071 [info] <0.803.0> accepting AMQP connection <0.803.0> (10.0.0.254:49370 -> 10.0.0.48:5673) 2025-10-20 05:14:55.079 [info] <0.803.0> connection <0.803.0> (10.0.0.254:49370 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:55.092 [info] <0.803.0> closing AMQP connection <0.803.0> (10.0.0.254:49370 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:14:55.093 [info] <0.816.0> accepting AMQP connection <0.816.0> (10.0.0.254:49380 -> 10.0.0.48:5673) 2025-10-20 05:14:55.097 [info] <0.816.0> connection <0.816.0> (10.0.0.254:49380 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:55.905 [info] <0.830.0> accepting AMQP connection <0.830.0> (10.0.0.254:49394 -> 10.0.0.48:5673) 2025-10-20 05:14:55.905 [info] <0.827.0> accepting AMQP connection <0.827.0> (10.0.0.254:49392 -> 10.0.0.48:5673) 2025-10-20 05:14:55.915 [info] <0.830.0> connection <0.830.0> (10.0.0.254:49394 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:55.915 [info] <0.827.0> connection <0.827.0> (10.0.0.254:49392 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:55.938 [info] <0.827.0> closing AMQP connection <0.827.0> (10.0.0.254:49392 -> 10.0.0.48:5673, vhost: '/', user: 'guest') 2025-10-20 05:14:55.939 [info] <0.859.0> accepting AMQP connection <0.859.0> (10.0.0.254:49400 -> 10.0.0.48:5673) 2025-10-20 05:14:55.941 [info] <0.859.0> connection <0.859.0> (10.0.0.254:49400 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:57.658 [info] <0.879.0> accepting AMQP connection <0.879.0> (10.0.0.49:49802 -> 10.0.0.48:5673) 2025-10-20 05:14:57.660 [info] <0.883.0> accepting AMQP connection <0.883.0> (10.0.0.49:49818 -> 10.0.0.48:5673) 2025-10-20 05:14:57.660 [info] <0.879.0> connection <0.879.0> (10.0.0.49:49802 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:57.661 [info] <0.891.0> accepting AMQP connection <0.891.0> (10.0.0.254:49416 -> 10.0.0.48:5673) 2025-10-20 05:14:57.662 [info] <0.894.0> accepting AMQP connection <0.894.0> (10.0.0.254:49426 -> 10.0.0.48:5673) 2025-10-20 05:14:57.662 [info] <0.897.0> accepting AMQP connection <0.897.0> (10.0.0.254:49428 -> 10.0.0.48:5673) 2025-10-20 05:14:57.663 [info] <0.883.0> connection <0.883.0> (10.0.0.49:49818 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:57.664 [info] <0.891.0> connection <0.891.0> (10.0.0.254:49416 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:57.664 [info] <0.894.0> connection <0.894.0> (10.0.0.254:49426 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:57.665 [info] <0.897.0> connection <0.897.0> (10.0.0.254:49428 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:57.665 [info] <0.928.0> accepting AMQP connection <0.928.0> (10.0.0.49:49832 -> 10.0.0.48:5673) 2025-10-20 05:14:57.665 [info] <0.934.0> accepting AMQP connection <0.934.0> (10.0.0.48:35178 -> 10.0.0.48:5673) 2025-10-20 05:14:57.666 [info] <0.944.0> accepting AMQP connection <0.944.0> (10.0.0.48:35192 -> 10.0.0.48:5673) 2025-10-20 05:14:57.667 [info] <0.941.0> accepting AMQP connection <0.941.0> (10.0.0.48:35190 -> 10.0.0.48:5673) 2025-10-20 05:14:57.668 [info] <0.944.0> connection <0.944.0> (10.0.0.48:35192 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:57.669 [info] <0.941.0> connection <0.941.0> (10.0.0.48:35190 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:57.670 [info] <0.928.0> connection <0.928.0> (10.0.0.49:49832 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:14:57.672 [info] <0.934.0> connection <0.934.0> (10.0.0.48:35178 -> 10.0.0.48:5673): user 'guest' authenticated and granted access to vhost '/' 2025-10-20 05:15:00.865 [info] <0.1042.0> RabbitMQ is asked to stop... 2025-10-20 05:15:00.959 [info] <0.1042.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:15:00.960 [info] <0.1042.0> Stopping application 'rabbitmq_management' 2025-10-20 05:15:00.963 [warning] <0.602.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:15:00.966 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:15:00.966 [info] <0.1042.0> Stopping application 'amqp_client' 2025-10-20 05:15:00.969 [info] <0.1042.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:15:00.969 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:15:00.970 [info] <0.1042.0> Stopping application 'cowboy' 2025-10-20 05:15:00.970 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:15:00.972 [info] <0.1042.0> Stopping application 'cowlib' 2025-10-20 05:15:00.972 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:15:00.972 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:15:00.972 [info] <0.1042.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:15:00.975 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:15:00.975 [info] <0.1042.0> Stopping application 'rabbit' 2025-10-20 05:15:00.976 [info] <0.246.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:15:00.976 [info] <0.594.0> stopped TCP listener on [::]:5673 2025-10-20 05:15:00.978 [error] <0.891.0> Error on AMQP connection <0.891.0> (10.0.0.254:49416 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.978 [error] <0.883.0> Error on AMQP connection <0.883.0> (10.0.0.49:49818 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.978 [error] <0.897.0> Error on AMQP connection <0.897.0> (10.0.0.254:49428 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.978 [error] <0.928.0> Error on AMQP connection <0.928.0> (10.0.0.49:49832 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.978 [error] <0.941.0> Error on AMQP connection <0.941.0> (10.0.0.48:35190 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.978 [error] <0.934.0> Error on AMQP connection <0.934.0> (10.0.0.48:35178 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.979 [error] <0.894.0> Error on AMQP connection <0.894.0> (10.0.0.254:49426 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.979 [error] <0.830.0> Error on AMQP connection <0.830.0> (10.0.0.254:49394 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.979 [error] <0.762.0> Error on AMQP connection <0.762.0> (10.0.0.48:35174 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.979 [error] <0.879.0> Error on AMQP connection <0.879.0> (10.0.0.49:49802 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.979 [error] <0.944.0> Error on AMQP connection <0.944.0> (10.0.0.48:35192 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.979 [error] <0.783.0> Error on AMQP connection <0.783.0> (10.0.0.254:49354 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.979 [error] <0.859.0> Error on AMQP connection <0.859.0> (10.0.0.254:49400 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.979 [error] <0.816.0> Error on AMQP connection <0.816.0> (10.0.0.254:49380 -> 10.0.0.48:5673, vhost: '/', user: 'guest', state: running), channel 0: operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'" 2025-10-20 05:15:00.982 [error] <0.917.0> Supervisor {<0.917.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error 2025-10-20 05:15:00.998 [info] <0.518.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:15:00.999 [warning] <0.1003.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:00.999 [warning] <0.996.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:00.999 [warning] <0.1007.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:00.999 [warning] <0.794.0> Mirrored queue 'vnc_config.cn-jenkins-deploy-platform-ansible-os-4371-1.-8082-0' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:00.999 [warning] <0.983.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:01.000 [warning] <0.774.0> Mirrored queue 'device_manager.cn-jenkins-deploy-platform-ansible-os-4371-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:01.000 [warning] <0.962.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-4371-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:01.000 [warning] <0.990.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:01.001 [warning] <0.932.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:01.001 [warning] <0.906.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-4371-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:01.001 [warning] <0.948.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-4371-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available 2025-10-20 05:15:01.005 [info] <0.548.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' 2025-10-20 05:15:01.044 [info] <0.548.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped 2025-10-20 05:15:01.044 [info] <0.545.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' 2025-10-20 05:15:01.048 [info] <0.545.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped 2025-10-20 05:15:01.056 [info] <0.1042.0> Stopping application 'mnesia' 2025-10-20 05:15:01.056 [info] <0.43.0> Application rabbit exited with reason: stopped 2025-10-20 05:15:01.060 [info] <0.1042.0> Stopping application 'rabbit_common' 2025-10-20 05:15:01.060 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:15:01.060 [info] <0.1042.0> Stopping application 'sysmon_handler' 2025-10-20 05:15:01.060 [info] <0.43.0> Application rabbit_common exited with reason: stopped 2025-10-20 05:15:01.061 [info] <0.1042.0> Stopping application 'os_mon' 2025-10-20 05:15:01.061 [info] <0.43.0> Application sysmon_handler exited with reason: stopped 2025-10-20 05:15:01.063 [info] <0.1042.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:15:01.063 [info] <0.43.0> Application os_mon exited with reason: stopped 2025-10-20 05:17:48.614 [info] <0.1142.0> Resetting Rabbit forcefully 2025-10-20 05:17:50.546 [info] <0.1156.0> RabbitMQ is asked to stop... 2025-10-20 05:17:50.611 [info] <0.1156.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_management_agent rabbitmq_web_dispatch rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:17:50.611 [info] <0.1156.0> Stopping application 'rabbitmq_management' 2025-10-20 05:17:50.611 [info] <0.1156.0> Stopping application 'amqp_client' 2025-10-20 05:17:50.611 [info] <0.1156.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:17:50.611 [info] <0.1156.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:17:50.611 [info] <0.1156.0> Stopping application 'rabbit' 2025-10-20 05:17:50.611 [info] <0.1156.0> Stopping application 'mnesia' 2025-10-20 05:17:50.611 [info] <0.1156.0> Stopping application 'rabbit_common' 2025-10-20 05:17:50.612 [info] <0.1156.0> Stopping application 'sysmon_handler' 2025-10-20 05:17:50.612 [info] <0.1156.0> Stopping application 'os_mon' 2025-10-20 05:17:50.612 [info] <0.1156.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:17:50.612 [info] <0.1156.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 observer_cli tools credentials_obfuscation crypto stdout_formatter recon inets jsx xmerl lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:17:57.280 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:17:59.373 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:17:59.374 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:17:59.392 [info] <0.240.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:17:59.393 [info] <0.240.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:17:59.405 [info] <0.240.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:17:59.405 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:17:59.405 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:17:59.410 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:17:59.418 [info] <0.248.0> Enabling free disk space monitoring 2025-10-20 05:17:59.418 [info] <0.248.0> Disk free limit set to 50MB 2025-10-20 05:17:59.424 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:17:59.424 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:17:59.424 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:17:59.424 [info] <0.252.0> FHC read buffering: OFF 2025-10-20 05:17:59.424 [info] <0.252.0> FHC write buffering: ON 2025-10-20 05:17:59.426 [info] <0.240.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:17:59.426 [info] <0.241.0> Will use 4 processes for default worker pool 2025-10-20 05:17:59.426 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:17:59.426 [info] <0.240.0> Running boot step database defined by app rabbit 2025-10-20 05:17:59.427 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:17:59.427 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:17:59.427 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:17:59.427 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:17:59.427 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:17:59.427 [info] <0.240.0> Will wait for 34179 milliseconds before proceeding with registration... 2025-10-20 05:18:33.607 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:18:33.607 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:18:33.612 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:18:33.616 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:18:33.616 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:18:34.118 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:18:34.119 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:18:34.119 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:18:34.621 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:18:34.623 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:18:34.623 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:18:35.125 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:18:35.126 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:18:35.126 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:18:35.628 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:18:35.629 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:18:35.629 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:18:36.131 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:18:36.132 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:18:36.132 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:18:36.635 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:18:36.636 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:18:36.636 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:18:37.138 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:18:37.138 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:18:37.138 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:18:37.642 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:18:37.645 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:18:37.645 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:18:38.147 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:18:38.151 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:18:38.151 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:18:38.652 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:18:38.653 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:18:38.773 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:18:38.773 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:18:38.807 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:18:38.807 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:18:38.838 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:18:38.838 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:18:38.838 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:18:38.838 [info] <0.240.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:18:38.838 [info] <0.240.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:18:38.838 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:18:38.838 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:18:38.838 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:18:38.838 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:18:38.838 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:18:38.838 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:18:38.839 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:18:38.840 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:18:38.840 [info] <0.468.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:18:38.841 [info] <0.240.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:18:38.844 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:18:38.844 [info] <0.472.0> Starting rabbit_node_monitor 2025-10-20 05:18:38.844 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:18:38.845 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:18:38.845 [info] <0.240.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:18:38.845 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:18:38.868 [info] <0.240.0> message_store upgrades: 1 to apply 2025-10-20 05:18:38.868 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:18:38.868 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:18:38.868 [info] <0.240.0> message_store upgrades: Removing the old message store data 2025-10-20 05:18:38.868 [info] <0.240.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:18:38.897 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:18:38.897 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:18:38.897 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:18:38.897 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:18:38.898 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:18:38.898 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:18:38.898 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:18:38.898 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:18:38.898 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:18:38.899 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:18:38.899 [warning] <0.240.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:18:38.899 [info] <0.240.0> Management plugin: using rates mode 'basic' 2025-10-20 05:18:38.899 [info] <0.240.0> Running boot step recovery defined by app rabbit 2025-10-20 05:18:38.899 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:18:38.900 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:18:38.900 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:18:38.900 [info] <0.240.0> Importing users... 2025-10-20 05:18:38.900 [info] <0.240.0> Creating user 'guest' 2025-10-20 05:18:38.906 [info] <0.240.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:18:38.909 [info] <0.240.0> Importing vhosts... 2025-10-20 05:18:38.909 [info] <0.240.0> Adding vhost '/' 2025-10-20 05:18:38.930 [info] <0.520.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:18:38.936 [info] <0.520.0> Starting message stores for vhost '/' 2025-10-20 05:18:38.936 [info] <0.524.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:18:38.937 [info] <0.520.0> Started message store of type transient for vhost '/' 2025-10-20 05:18:38.937 [info] <0.527.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:18:38.938 [warning] <0.527.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:18:38.938 [info] <0.520.0> Started message store of type persistent for vhost '/' 2025-10-20 05:18:38.939 [info] <0.240.0> Importing user permissions... 2025-10-20 05:18:38.939 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:18:38.943 [info] <0.240.0> Importing topic permissions... 2025-10-20 05:18:38.943 [info] <0.240.0> Importing parameters... 2025-10-20 05:18:38.943 [info] <0.240.0> Importing global parameters... 2025-10-20 05:18:38.943 [info] <0.240.0> Importing policies... 2025-10-20 05:18:38.951 [info] <0.240.0> Importing queues... 2025-10-20 05:18:38.952 [info] <0.240.0> Importing exchanges... 2025-10-20 05:18:38.952 [info] <0.240.0> Importing bindings... 2025-10-20 05:18:38.952 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:18:38.952 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:18:38.952 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:18:38.953 [info] <0.240.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:18:38.954 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:18:38.959 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:18:38.964 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:18:38.964 [info] <0.240.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:18:38.964 [info] <0.240.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:18:38.964 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:18:38.964 [info] <0.240.0> Running boot step networking defined by app rabbit 2025-10-20 05:18:38.968 [info] <0.573.0> started TCP listener on [::]:5673 2025-10-20 05:18:38.969 [info] <0.240.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:18:38.969 [info] <0.240.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:18:38.970 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:18:38.970 [info] <0.575.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:18:39.004 [info] <0.625.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:18:39.004 [info] <0.731.0> Statistics database started. 2025-10-20 05:18:39.004 [info] <0.730.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:18:39.005 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:18:39.115 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch 2025-10-20 05:18:41.120 [info] <0.755.0> RabbitMQ is asked to stop... 2025-10-20 05:18:41.228 [info] <0.755.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:18:41.228 [info] <0.755.0> Stopping application 'rabbitmq_management' 2025-10-20 05:18:41.229 [warning] <0.581.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:18:41.231 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:18:41.231 [info] <0.755.0> Stopping application 'amqp_client' 2025-10-20 05:18:41.232 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:18:41.232 [info] <0.755.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:18:41.233 [info] <0.755.0> Stopping application 'cowboy' 2025-10-20 05:18:41.233 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:18:41.234 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:18:41.234 [info] <0.755.0> Stopping application 'cowlib' 2025-10-20 05:18:41.234 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:18:41.234 [info] <0.755.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:18:41.236 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:18:41.236 [info] <0.755.0> Stopping application 'rabbit' 2025-10-20 05:18:41.236 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:18:41.236 [info] <0.573.0> stopped TCP listener on [::]:5673 2025-10-20 05:18:41.237 [info] <0.527.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' 2025-10-20 05:18:41.237 [info] <0.497.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:18:41.242 [info] <0.527.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped 2025-10-20 05:18:41.242 [info] <0.524.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' 2025-10-20 05:18:41.245 [info] <0.524.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped 2025-10-20 05:18:41.248 [info] <0.43.0> Application rabbit exited with reason: stopped 2025-10-20 05:18:41.248 [info] <0.755.0> Stopping application 'mnesia' 2025-10-20 05:18:41.250 [info] <0.755.0> Stopping application 'rabbit_common' 2025-10-20 05:18:41.250 [info] <0.755.0> Stopping application 'sysmon_handler' 2025-10-20 05:18:41.250 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:18:41.250 [info] <0.43.0> Application rabbit_common exited with reason: stopped 2025-10-20 05:18:41.251 [info] <0.755.0> Stopping application 'os_mon' 2025-10-20 05:18:41.251 [info] <0.43.0> Application sysmon_handler exited with reason: stopped 2025-10-20 05:18:41.252 [info] <0.755.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:18:41.252 [info] <0.43.0> Application os_mon exited with reason: stopped 2025-10-20 05:21:06.422 [info] <0.793.0> Resetting Rabbit forcefully 2025-10-20 05:21:08.387 [info] <0.807.0> RabbitMQ is asked to stop... 2025-10-20 05:21:08.461 [info] <0.807.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_management_agent rabbitmq_web_dispatch rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:21:08.461 [info] <0.807.0> Stopping application 'rabbitmq_management' 2025-10-20 05:21:08.461 [info] <0.807.0> Stopping application 'amqp_client' 2025-10-20 05:21:08.461 [info] <0.807.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:21:08.461 [info] <0.807.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:21:08.461 [info] <0.807.0> Stopping application 'rabbit' 2025-10-20 05:21:08.461 [info] <0.807.0> Stopping application 'mnesia' 2025-10-20 05:21:08.461 [info] <0.807.0> Stopping application 'rabbit_common' 2025-10-20 05:21:08.461 [info] <0.807.0> Stopping application 'sysmon_handler' 2025-10-20 05:21:08.461 [info] <0.807.0> Stopping application 'os_mon' 2025-10-20 05:21:08.461 [info] <0.807.0> Successfully stopped RabbitMQ and its dependencies 2025-10-20 05:21:08.462 [info] <0.807.0> Halting Erlang VM with the following applications: ranch ssl public_key asn1 observer_cli tools credentials_obfuscation crypto stdout_formatter recon inets jsx xmerl lager goldrush compiler syntax_tools sasl stdlib kernel 2025-10-20 05:21:14.154 [info] <0.8.0> Log file opened with Lager 2025-10-20 05:21:15.950 [info] <0.8.0> Feature flags: list of feature flags found: 2025-10-20 05:21:15.950 [info] <0.8.0> Feature flags: feature flag states written to disk: yes 2025-10-20 05:21:15.968 [info] <0.240.0> Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21 Copyright (c) 2007-2020 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ 2025-10-20 05:21:15.969 [info] <0.240.0> node : contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : esZsDxSN6VGbi9JkMSxNZA== log(s) : /var/log/contrail/config-database-rabbitmq/rabbitmq.log : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2_upgrade.log database dir : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 2025-10-20 05:21:15.980 [info] <0.240.0> Running boot step pre_boot defined by app rabbit 2025-10-20 05:21:15.980 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit 2025-10-20 05:21:15.980 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit 2025-10-20 05:21:15.989 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total 2025-10-20 05:21:15.994 [info] <0.248.0> Enabling free disk space monitoring 2025-10-20 05:21:15.994 [info] <0.248.0> Disk free limit set to 50MB 2025-10-20 05:21:16.000 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit 2025-10-20 05:21:16.000 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit 2025-10-20 05:21:16.000 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets) 2025-10-20 05:21:16.000 [info] <0.252.0> FHC read buffering: OFF 2025-10-20 05:21:16.000 [info] <0.252.0> FHC write buffering: ON 2025-10-20 05:21:16.000 [info] <0.240.0> Running boot step worker_pool defined by app rabbit 2025-10-20 05:21:16.000 [info] <0.241.0> Will use 4 processes for default worker pool 2025-10-20 05:21:16.000 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it 2025-10-20 05:21:16.000 [info] <0.240.0> Running boot step database defined by app rabbit 2025-10-20 05:21:16.001 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2025-10-20 05:21:16.001 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config 2025-10-20 05:21:16.001 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:21:16.001 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay 2025-10-20 05:21:16.001 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration. 2025-10-20 05:21:16.001 [info] <0.240.0> Will wait for 49403 milliseconds before proceeding with registration... 2025-10-20 05:22:01.865 [info] <0.307.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping... 2025-10-20 05:22:05.404 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-2, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:22:05.405 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1, contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 2025-10-20 05:22:05.405 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:22:05.407 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:22:05.407 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left... 2025-10-20 05:22:05.908 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {error,mnesia_not_running} 2025-10-20 05:22:05.909 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:22:05.909 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left... 2025-10-20 05:22:06.412 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:22:06.413 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:22:06.414 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left... 2025-10-20 05:22:06.917 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:22:06.918 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:22:06.918 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left... 2025-10-20 05:22:07.421 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:22:07.421 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:22:07.421 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left... 2025-10-20 05:22:07.923 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:22:07.924 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:22:07.924 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left... 2025-10-20 05:22:08.426 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:22:08.427 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:22:08.427 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left... 2025-10-20 05:22:08.929 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:22:08.930 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:22:08.930 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left... 2025-10-20 05:22:09.432 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:22:09.433 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:22:09.434 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left... 2025-10-20 05:22:09.935 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-1: {badrpc,nodedown} 2025-10-20 05:22:09.936 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-4371-3: {error,mnesia_not_running} 2025-10-20 05:22:09.936 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left... 2025-10-20 05:22:10.437 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-4371-1,contrail@cn-jenkins-deploy-platform-ansible-os-4371-3 (as in Erlang distribution). Starting as a blank standalone node... 2025-10-20 05:22:10.440 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:22:10.562 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:22:10.562 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:22:10.590 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:22:10.590 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:22:10.620 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2025-10-20 05:22:10.620 [info] <0.240.0> Successfully synced tables from a peer 2025-10-20 05:22:10.620 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:22:10.620 [info] <0.240.0> Running boot step database_sync defined by app rabbit 2025-10-20 05:22:10.620 [info] <0.240.0> Running boot step feature_flags defined by app rabbit 2025-10-20 05:22:10.620 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-20 05:22:10.620 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit 2025-10-20 05:22:10.620 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit 2025-10-20 05:22:10.620 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit 2025-10-20 05:22:10.621 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-20 05:22:10.622 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-20 05:22:10.622 [info] <0.487.0> epmd monitor knows us, inter-node communication (distribution) port: 25673 2025-10-20 05:22:10.622 [info] <0.240.0> Running boot step guid_generator defined by app rabbit 2025-10-20 05:22:10.628 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-20 05:22:10.628 [info] <0.491.0> Starting rabbit_node_monitor 2025-10-20 05:22:10.629 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit 2025-10-20 05:22:10.629 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit 2025-10-20 05:22:10.629 [info] <0.240.0> Running boot step core_initialized defined by app rabbit 2025-10-20 05:22:10.629 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit 2025-10-20 05:22:10.653 [info] <0.240.0> message_store upgrades: 1 to apply 2025-10-20 05:22:10.654 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store 2025-10-20 05:22:10.654 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration 2025-10-20 05:22:10.654 [info] <0.240.0> message_store upgrades: Removing the old message store data 2025-10-20 05:22:10.654 [info] <0.240.0> message_store upgrades: All upgrades applied successfully 2025-10-20 05:22:10.681 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit 2025-10-20 05:22:10.682 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-20 05:22:10.682 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-20 05:22:10.682 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit 2025-10-20 05:22:10.683 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit 2025-10-20 05:22:10.683 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit 2025-10-20 05:22:10.683 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit 2025-10-20 05:22:10.683 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-20 05:22:10.683 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-20 05:22:10.684 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-20 05:22:10.684 [warning] <0.240.0> force_fine_statistics set to true; ignored. Replaced by {rates_mode, none} in the rabbitmq_management application. 2025-10-20 05:22:10.684 [info] <0.240.0> Management plugin: using rates mode 'basic' 2025-10-20 05:22:10.684 [info] <0.240.0> Running boot step recovery defined by app rabbit 2025-10-20 05:22:10.684 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management 2025-10-20 05:22:10.685 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json 2025-10-20 05:22:10.685 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal 2025-10-20 05:22:10.685 [info] <0.240.0> Importing users... 2025-10-20 05:22:10.685 [info] <0.240.0> Creating user 'guest' 2025-10-20 05:22:10.692 [info] <0.240.0> Setting user tags for user 'guest' to [administrator] 2025-10-20 05:22:10.695 [info] <0.240.0> Importing vhosts... 2025-10-20 05:22:10.695 [info] <0.240.0> Adding vhost '/' 2025-10-20 05:22:10.722 [info] <0.539.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-20 05:22:10.730 [info] <0.539.0> Starting message stores for vhost '/' 2025-10-20 05:22:10.730 [info] <0.543.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:22:10.731 [info] <0.539.0> Started message store of type transient for vhost '/' 2025-10-20 05:22:10.731 [info] <0.546.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index 2025-10-20 05:22:10.732 [warning] <0.546.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-20 05:22:10.732 [info] <0.539.0> Started message store of type persistent for vhost '/' 2025-10-20 05:22:10.733 [info] <0.240.0> Importing user permissions... 2025-10-20 05:22:10.734 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*' 2025-10-20 05:22:10.737 [info] <0.240.0> Importing topic permissions... 2025-10-20 05:22:10.737 [info] <0.240.0> Importing parameters... 2025-10-20 05:22:10.737 [info] <0.240.0> Importing global parameters... 2025-10-20 05:22:10.737 [info] <0.240.0> Importing policies... 2025-10-20 05:22:10.743 [info] <0.240.0> Importing queues... 2025-10-20 05:22:10.743 [info] <0.240.0> Importing exchanges... 2025-10-20 05:22:10.744 [info] <0.240.0> Importing bindings... 2025-10-20 05:22:10.744 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit 2025-10-20 05:22:10.744 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit 2025-10-20 05:22:10.744 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-20 05:22:10.744 [info] <0.240.0> Running boot step background_gc defined by app rabbit 2025-10-20 05:22:10.744 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit 2025-10-20 05:22:10.749 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:22:10.754 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' 2025-10-20 05:22:10.754 [info] <0.240.0> Running boot step routing_ready defined by app rabbit 2025-10-20 05:22:10.754 [info] <0.240.0> Running boot step pre_flight defined by app rabbit 2025-10-20 05:22:10.754 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit 2025-10-20 05:22:10.754 [info] <0.240.0> Running boot step networking defined by app rabbit 2025-10-20 05:22:10.756 [info] <0.592.0> started TCP listener on [::]:5673 2025-10-20 05:22:10.757 [info] <0.240.0> Running boot step cluster_name defined by app rabbit 2025-10-20 05:22:10.757 [info] <0.240.0> Running boot step direct_client defined by app rabbit 2025-10-20 05:22:10.757 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit 2025-10-20 05:22:10.757 [info] <0.594.0> Swapping OS signal event handler (erl_signal_server) for our own 2025-10-20 05:22:10.787 [info] <0.644.0> Management plugin: HTTP (non-TLS) listener started on port 15673 2025-10-20 05:22:10.787 [info] <0.750.0> Statistics database started. 2025-10-20 05:22:10.787 [info] <0.749.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-20 05:22:10.787 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50 2025-10-20 05:22:10.887 [info] <0.8.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch 2025-10-20 05:22:10.956 [info] <0.307.0> RabbitMQ is asked to stop... 2025-10-20 05:22:10.998 [info] <0.307.0> Stopping RabbitMQ applications and their dependencies in the following order: rabbitmq_management amqp_client rabbitmq_web_dispatch cowboy cowlib rabbitmq_management_agent rabbit mnesia rabbit_common sysmon_handler os_mon 2025-10-20 05:22:10.999 [info] <0.307.0> Stopping application 'rabbitmq_management' 2025-10-20 05:22:11.000 [warning] <0.600.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls 2025-10-20 05:22:11.001 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped 2025-10-20 05:22:11.001 [info] <0.307.0> Stopping application 'amqp_client' 2025-10-20 05:22:11.003 [info] <0.43.0> Application amqp_client exited with reason: stopped 2025-10-20 05:22:11.003 [info] <0.307.0> Stopping application 'rabbitmq_web_dispatch' 2025-10-20 05:22:11.004 [info] <0.307.0> Stopping application 'cowboy' 2025-10-20 05:22:11.004 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped 2025-10-20 05:22:11.005 [info] <0.43.0> Application cowboy exited with reason: stopped 2025-10-20 05:22:11.005 [info] <0.307.0> Stopping application 'cowlib' 2025-10-20 05:22:11.005 [info] <0.43.0> Application cowlib exited with reason: stopped 2025-10-20 05:22:11.005 [info] <0.307.0> Stopping application 'rabbitmq_management_agent' 2025-10-20 05:22:11.007 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped 2025-10-20 05:22:11.007 [info] <0.307.0> Stopping application 'rabbit' 2025-10-20 05:22:11.007 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config 2025-10-20 05:22:11.007 [info] <0.592.0> stopped TCP listener on [::]:5673 2025-10-20 05:22:11.007 [info] <0.516.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-4371-2' because the vhost is stopping 2025-10-20 05:22:11.008 [info] <0.546.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' 2025-10-20 05:22:11.013 [info] <0.546.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped 2025-10-20 05:22:11.014 [info] <0.543.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' 2025-10-20 05:22:11.019 [info] <0.543.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-4371-2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped 2025-10-20 05:22:11.021 [info] <0.43.0> Application rabbit exited with reason: stopped 2025-10-20 05:22:11.021 [info] <0.307.0> Stopping application 'mnesia' 2025-10-20 05:22:11.027 [info] <0.43.0> Application mnesia exited with reason: stopped 2025-10-20 05:22:11.028 [info] <0.307.0> Stopping application 'rabbit_common' 2025-10-20 05:22:11.028 [info] <0.43.0> Application rabbit_common exited with reason: stopped 2025-10-20 05:22:11.028 [info] <0.307.0> Stopping application 'sysmon_handler' 2025-10-20 05:22:11.029 [info] <0.43.0> Application sysmon_handler exited with reason: stopped 2025-10-20 05:22:11.029 [info] <0.307.0> Stopping application 'os_mon' 2025-10-20 05:22:11.030 [info] <0.43.0> Application os_mon exited with reason: stopped 2025-10-20 05:22:11.030 [info] <0.307.0> Successfully stopped RabbitMQ and its dependencies