07/06/2026 14:37:49.048 7f36a34e44a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/06/2026 14:37:49.048 7f36a34e44a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 107 start_time = 1783348669047860 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 07/06/2026 14:37:49.072 7f36a7f0c800 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 07/06/2026 14:37:49.080 7f36a7f0c800 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/06/2026 14:37:49.081 7f36a34e4e00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/06/2026 14:37:49.081 7f36a34e4e00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/06/2026 14:37:49.093 7f36a7f0c800 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/06/2026 14:37:49.095 7f36a34e4f40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/06/2026 14:37:49.124 7f36a7f0c800 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/06/2026 14:37:49.124 7f36a374e720 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/06/2026 14:37:49.137 7f36a7f0c800 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/06/2026 14:37:49.137 7f36a374e720 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/06/2026 14:37:49.162 7f36a7f0c800 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 07/06/2026 14:37:49.163 7f36a7f0c800 [contrail-device-manager] [WARNING]: initializing cassandra cluster 07/06/2026 14:37:49.163 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 07/06/2026 14:37:49.174 7f36a374ee00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/06/2026 14:37:49.175 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 07/06/2026 14:37:49.230 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 07/06/2026 14:37:49.231 7f36a38afa40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 07/06/2026 14:37:49.245 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 07/06/2026 14:37:49.245 7f36a38af9a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/06/2026 14:37:49.273 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 07/06/2026 14:37:49.470 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 07/06/2026 14:37:49.471 7f36a3555720 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/06/2026 14:37:49.507 7f36a374e900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 07/06/2026 14:37:49.524 7f36a374e900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/06/2026 14:37:50.028 7f36a7f0c800 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 07/06/2026 14:37:50.529 7f36a7f0c800 [contrail-device-manager] [WARNING]: Process 107 prepared to run in Full mode... 07/06/2026 14:37:50.529 7f36a7f0c800 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 07/06/2026 14:37:50.596 7f36a37cd860 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 07/06/2026 14:37:50.596 7f36a37cd860 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 07/06/2026 14:37:50.828 7f36a36117c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/06/2026 14:37:50.835 7f36a3611ae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/06/2026 14:37:53.049 7f36a3611d60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-421-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/06/2026 14:37:53.868 7f36a7f0c800 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 07/06/2026 14:37:53.868 7f36a7f0c800 [contrail-device-manager] [ERROR]: Unable to load init data 07/06/2026 14:37:53.868 7f36a7f0c800 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 07/06/2026 14:37:53.868 7f36a7f0c800 [contrail-device-manager] [WARNING]: initializing cassandra cluster 07/06/2026 14:37:53.868 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 07/06/2026 14:37:53.874 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 07/06/2026 14:37:53.954 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 07/06/2026 14:37:53.968 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 07/06/2026 14:37:53.999 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/06/2026 14:37:54.116 7f36a7f0c800 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 07/06/2026 14:37:54.122 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/06/2026 14:37:54.134 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/06/2026 14:37:54.206 7f36a7f0c800 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 07/06/2026 14:37:54.209 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/06/2026 14:37:54.209 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 07/06/2026 14:37:54.243 7f36a7f0c800 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 07/06/2026 14:37:54.251 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/06/2026 14:37:54.467 7f36a7f0c800 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 07/06/2026 14:37:54.473 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/06/2026 14:37:54.481 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/06/2026 14:37:54.592 7f36a7f0c800 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 07/06/2026 14:37:54.595 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/06/2026 14:37:54.600 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/06/2026 14:37:54.756 7f36a7f0c800 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 07/06/2026 14:37:54.759 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/06/2026 14:37:54.763 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/06/2026 14:37:54.843 7f36a7f0c800 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 07/06/2026 14:37:54.846 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/06/2026 14:37:54.859 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/06/2026 14:37:54.987 7f36a7f0c800 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 07/06/2026 14:37:54.992 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/06/2026 14:37:54.996 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/06/2026 14:37:55.139 7f36a7f0c800 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 07/06/2026 14:37:55.142 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/06/2026 14:37:55.149 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/06/2026 14:37:55.339 7f36a7f0c800 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 07/06/2026 14:37:55.343 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/06/2026 14:37:55.351 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/06/2026 14:37:55.503 7f36a7f0c800 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 07/06/2026 14:37:55.515 7f36a7f0c800 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/06/2026 14:37:55.549 7f36a7f0c800 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 07/06/2026 14:37:55.549 7f36a7f0c800 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/06/2026 14:37:55.556 7f36a7f0c800 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/06/2026 14:37:55.576 7f36a7f0c800 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/06/2026 14:37:55.590 7f36a7f0c800 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/06/2026 14:37:55.592 7f36a2daaea0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/06/2026 14:37:55.624 7f36a2daaea0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/06/2026 14:37:55.854 7f36a2ba90e0 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 07/06/2026 14:37:55.854 7f36a2ba90e0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/06/2026 14:37:55.875 7f36a2ba90e0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/06/2026 14:40:34.119 7f36a2ba90e0 [contrail-device-manager] [WARNING]: instance_ip uuid 9baab0fb-3978-4a16-bbc5-2d38759c6cea update caused NoIdError