05/28/2026 01:38:34.030 7f61ca65c860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/28/2026 01:38:34.031 7f61ca65c860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 105 start_time = 1779932314030747 collector_name = collector_ip = collector_list = [ 10.0.0.241:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 05/28/2026 01:38:34.062 7f61cf1ae980 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.241:5673//'] 05/28/2026 01:38:34.070 7f61cf1ae980 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/28/2026 01:38:34.071 7f61ca915220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.241:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/28/2026 01:38:34.071 7f61ca915220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.241:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/28/2026 01:38:34.096 7f61cf1ae980 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/28/2026 01:38:34.096 7f61ca915360 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.241:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/28/2026 01:38:34.252 7f61cf1ae980 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/28/2026 01:38:34.252 7f61ca915ae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.241:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/28/2026 01:38:34.277 7f61cf1ae980 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 05/28/2026 01:38:34.277 7f61ca915ae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.241:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/28/2026 01:38:34.392 7f61cf1ae980 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 05/28/2026 01:38:34.392 7f61cf1ae980 [contrail-device-manager] [WARNING]: initializing cassandra cluster 05/28/2026 01:38:34.393 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 05/28/2026 01:38:34.393 7f61ca5b5220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.241:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.241:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 05/28/2026 01:38:34.422 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 05/28/2026 01:38:35.220 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 05/28/2026 01:38:35.230 7f61ca5b5e00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.241:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.241:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.241:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 05/28/2026 01:38:35.258 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 05/28/2026 01:38:35.269 7f61ca5b5d60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.241:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.241:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.241:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 05/28/2026 01:38:35.309 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 05/28/2026 01:38:35.503 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 05/28/2026 01:38:35.579 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:36.247 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 05/28/2026 01:38:36.261 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:36.956 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:37.187 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 05/28/2026 01:38:37.196 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:38.033 7f61ca915220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-413-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.241:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.241:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.241:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.241:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 05/28/2026 01:38:38.281 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:38.283 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 05/28/2026 01:38:38.288 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:38.605 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:38.717 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 05/28/2026 01:38:38.723 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:39.245 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:39.249 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 05/28/2026 01:38:39.256 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:39.635 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:39.727 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 05/28/2026 01:38:39.732 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:40.261 7f61cf1ae980 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 05/28/2026 01:38:40.763 7f61cf1ae980 [contrail-device-manager] [WARNING]: Process 105 prepared to run in Full mode... 05/28/2026 01:38:40.763 7f61cf1ae980 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 05/28/2026 01:38:40.803 7f61ca9d6f40 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 05/28/2026 01:38:40.803 7f61ca9d6f40 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 05/28/2026 01:38:44.085 7f61cf1ae980 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 05/28/2026 01:38:44.085 7f61cf1ae980 [contrail-device-manager] [ERROR]: Unable to load init data 05/28/2026 01:38:44.086 7f61cf1ae980 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 05/28/2026 01:38:44.086 7f61cf1ae980 [contrail-device-manager] [WARNING]: initializing cassandra cluster 05/28/2026 01:38:44.086 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 05/28/2026 01:38:44.088 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 05/28/2026 01:38:44.130 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 05/28/2026 01:38:44.199 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 05/28/2026 01:38:44.481 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:44.607 7f61cf1ae980 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 05/28/2026 01:38:44.629 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:44.898 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:44.955 7f61cf1ae980 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 05/28/2026 01:38:44.958 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:44.959 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 05/28/2026 01:38:45.459 7f61cf1ae980 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 05/28/2026 01:38:45.486 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:45.799 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 05/28/2026 01:38:45.812 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:45.833 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:46.087 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 05/28/2026 01:38:46.100 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:46.136 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:46.453 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 05/28/2026 01:38:46.463 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:46.473 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:46.721 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 05/28/2026 01:38:46.754 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:46.814 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:47.093 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 05/28/2026 01:38:47.099 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:47.123 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:47.238 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 05/28/2026 01:38:47.249 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:47.286 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:47.505 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 05/28/2026 01:38:47.512 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:47.528 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/28/2026 01:38:47.995 7f61cf1ae980 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 05/28/2026 01:38:47.999 7f61cf1ae980 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/28/2026 01:38:48.054 7f61cf1ae980 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.241:5673//'] 05/28/2026 01:38:48.054 7f61cf1ae980 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/28/2026 01:38:48.084 7f61cf1ae980 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/28/2026 01:38:48.100 7f61cf1ae980 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/28/2026 01:38:48.120 7f61cf1ae980 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 05/28/2026 01:38:48.128 7f61ca074ea0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/28/2026 01:38:48.178 7f61ca074ea0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED