06/22/2026 01:35:11.464 7fe627c584a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/22/2026 01:35:11.464 7fe627c584a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 112 start_time = 1782092111463866 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 06/22/2026 01:35:11.494 7fe62c364140 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//'] 06/22/2026 01:35:11.502 7fe62c364140 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 06/22/2026 01:35:11.502 7fe627c58e00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 06/22/2026 01:35:11.503 7fe627c58e00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 06/22/2026 01:35:11.506 7fe62c364140 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/22/2026 01:35:11.507 7fe627c58f40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/22/2026 01:35:11.526 7fe62c364140 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 06/22/2026 01:35:11.527 7fe627be6720 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 06/22/2026 01:35:11.536 7fe62c364140 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 06/22/2026 01:35:11.537 7fe627be6720 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/22/2026 01:35:11.558 7fe62c364140 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 06/22/2026 01:35:11.559 7fe62c364140 [contrail-device-manager] [WARNING]: initializing cassandra cluster 06/22/2026 01:35:11.559 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 06/22/2026 01:35:11.560 7fe627be6e00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 06/22/2026 01:35:11.561 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 06/22/2026 01:35:11.610 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 06/22/2026 01:35:11.611 7fe627b0ee00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 06/22/2026 01:35:11.624 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 06/22/2026 01:35:11.625 7fe627b0ed60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 06/22/2026 01:35:11.631 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 06/22/2026 01:35:11.671 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 06/22/2026 01:35:11.671 7fe627c8a720 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/22/2026 01:35:11.683 7fe627be6900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 06/22/2026 01:35:11.691 7fe627be6900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/22/2026 01:35:12.198 7fe62c364140 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 06/22/2026 01:35:12.707 7fe62c364140 [contrail-device-manager] [WARNING]: Process 112 prepared to run in Full mode... 06/22/2026 01:35:12.707 7fe62c364140 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 06/22/2026 01:35:12.788 7fe627bbb860 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 06/22/2026 01:35:12.788 7fe627bbb860 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 06/22/2026 01:35:13.084 7fe62714b7c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 06/22/2026 01:35:13.088 7fe62714ba40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 06/22/2026 01:35:15.472 7fe62714bea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-420-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 06/22/2026 01:35:16.112 7fe62c364140 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 06/22/2026 01:35:16.112 7fe62c364140 [contrail-device-manager] [ERROR]: Unable to load init data 06/22/2026 01:35:16.113 7fe62c364140 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 06/22/2026 01:35:16.113 7fe62c364140 [contrail-device-manager] [WARNING]: initializing cassandra cluster 06/22/2026 01:35:16.113 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 06/22/2026 01:35:16.121 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 06/22/2026 01:35:16.241 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 06/22/2026 01:35:16.257 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 06/22/2026 01:35:16.394 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 06/22/2026 01:35:16.568 7fe62c364140 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 06/22/2026 01:35:16.590 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock released. 06/22/2026 01:35:16.816 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 06/22/2026 01:35:16.966 7fe62c364140 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 06/22/2026 01:35:16.972 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock released. 06/22/2026 01:35:16.972 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 06/22/2026 01:35:17.067 7fe62c364140 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 06/22/2026 01:35:17.252 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 06/22/2026 01:35:17.596 7fe62c364140 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 06/22/2026 01:35:17.601 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock released. 06/22/2026 01:35:17.794 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 06/22/2026 01:35:18.057 7fe62c364140 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 06/22/2026 01:35:18.071 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock released. 06/22/2026 01:35:18.499 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 06/22/2026 01:35:18.937 7fe62c364140 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 06/22/2026 01:35:18.982 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock released. 06/22/2026 01:35:19.247 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 06/22/2026 01:35:19.444 7fe62c364140 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 06/22/2026 01:35:19.451 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock released. 06/22/2026 01:35:20.122 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 06/22/2026 01:35:20.301 7fe62c364140 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 06/22/2026 01:35:20.309 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock released. 06/22/2026 01:35:20.536 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 06/22/2026 01:35:20.722 7fe62c364140 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 06/22/2026 01:35:20.727 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock released. 06/22/2026 01:35:20.736 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 06/22/2026 01:35:20.996 7fe62c364140 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 06/22/2026 01:35:21.002 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock released. 06/22/2026 01:35:21.026 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 06/22/2026 01:35:21.140 7fe62c364140 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 06/22/2026 01:35:21.146 7fe62c364140 [contrail-device-manager] [WARNING]: Zookeeper lock released. 06/22/2026 01:35:21.167 7fe62c364140 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//'] 06/22/2026 01:35:21.168 7fe62c364140 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 06/22/2026 01:35:21.189 7fe62c364140 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/22/2026 01:35:21.200 7fe62c364140 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 06/22/2026 01:35:21.213 7fe62c364140 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 06/22/2026 01:35:21.215 7fe626be2a40 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 06/22/2026 01:35:21.236 7fe626be2a40 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED