04/27/2026 01:58:21.834 7f3033a04860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/27/2026 01:58:21.834 7f3033a04860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1777255101834232 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 04/27/2026 01:58:22.215 7f30381272c0 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//'] 04/27/2026 01:58:22.243 7f30381272c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 04/27/2026 01:58:22.244 7f3033730220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/27/2026 01:58:22.244 7f3033730220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/27/2026 01:58:22.254 7f30381272c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/27/2026 01:58:22.255 7f3033730360 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/27/2026 01:58:22.320 7f30381272c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/27/2026 01:58:22.320 7f3033730ae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/27/2026 01:58:22.341 7f30381272c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/27/2026 01:58:22.342 7f3033730ae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/27/2026 01:58:22.381 7f30381272c0 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 04/27/2026 01:58:22.381 7f30381272c0 [contrail-device-manager] [WARNING]: initializing cassandra cluster 04/27/2026 01:58:22.392 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 04/27/2026 01:58:22.393 7f30336b3220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 04/27/2026 01:58:22.408 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 04/27/2026 01:58:22.687 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 04/27/2026 01:58:22.688 7f30336b3a40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 04/27/2026 01:58:22.716 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 04/27/2026 01:58:22.716 7f30336b39a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 04/27/2026 01:58:22.759 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 04/27/2026 01:58:22.907 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 04/27/2026 01:58:22.915 7f3033a53ae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/27/2026 01:58:22.923 7f3033730cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 04/27/2026 01:58:23.015 7f3033730cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/27/2026 01:58:23.527 7f30381272c0 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 04/27/2026 01:58:24.042 7f30381272c0 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 04/27/2026 01:58:24.042 7f30381272c0 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 04/27/2026 01:58:24.112 7f3033972f40 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 04/27/2026 01:58:24.112 7f3033972f40 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 04/27/2026 01:58:25.023 7f3032e52cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 04/27/2026 01:58:25.072 7f3032e52d60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 04/27/2026 01:58:25.845 7f3032e52f40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-404-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 04/27/2026 01:58:28.082 7f30381272c0 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 04/27/2026 01:58:28.082 7f30381272c0 [contrail-device-manager] [ERROR]: Unable to load init data 04/27/2026 01:58:28.082 7f30381272c0 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 04/27/2026 01:58:28.082 7f30381272c0 [contrail-device-manager] [WARNING]: initializing cassandra cluster 04/27/2026 01:58:28.084 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 04/27/2026 01:58:28.096 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 04/27/2026 01:58:28.363 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 04/27/2026 01:58:28.453 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 04/27/2026 01:58:28.611 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/27/2026 01:58:28.819 7f30381272c0 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 04/27/2026 01:58:28.830 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/27/2026 01:58:29.000 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/27/2026 01:58:29.079 7f30381272c0 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 04/27/2026 01:58:29.100 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/27/2026 01:58:29.100 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 04/27/2026 01:58:29.239 7f30381272c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 04/27/2026 01:58:29.356 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/27/2026 01:58:29.585 7f30381272c0 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 04/27/2026 01:58:29.590 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/27/2026 01:58:29.758 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/27/2026 01:58:29.942 7f30381272c0 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 04/27/2026 01:58:29.954 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/27/2026 01:58:30.445 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/27/2026 01:58:30.853 7f30381272c0 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 04/27/2026 01:58:30.863 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/27/2026 01:58:31.512 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/27/2026 01:58:31.869 7f30381272c0 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 04/27/2026 01:58:31.879 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/27/2026 01:58:33.000 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/27/2026 01:58:33.511 7f30381272c0 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 04/27/2026 01:58:33.527 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/27/2026 01:58:35.139 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/27/2026 01:58:35.344 7f30381272c0 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 04/27/2026 01:58:35.370 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/27/2026 01:58:35.691 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/27/2026 01:58:35.997 7f30381272c0 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 04/27/2026 01:58:36.003 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/27/2026 01:58:36.220 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/27/2026 01:58:36.454 7f30381272c0 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 04/27/2026 01:58:36.460 7f30381272c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/27/2026 01:58:36.510 7f30381272c0 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//'] 04/27/2026 01:58:36.511 7f30381272c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 04/27/2026 01:58:36.528 7f30381272c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/27/2026 01:58:36.557 7f30381272c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/27/2026 01:58:36.571 7f30381272c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/27/2026 01:58:36.574 7f3032ff4720 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/27/2026 01:58:36.593 7f3032ff4720 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED