05/04/2026 01:52:14.629 7fc459492860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/04/2026 01:52:14.629 7fc459492860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 109 start_time = 1777859534629071 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 05/04/2026 01:52:14.658 7fc45dee2180 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//'] 05/04/2026 01:52:14.667 7fc45dee2180 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/04/2026 01:52:14.667 7fc45977e220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/04/2026 01:52:14.668 7fc45977e220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/04/2026 01:52:14.683 7fc45dee2180 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/04/2026 01:52:14.683 7fc45977e360 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/04/2026 01:52:14.714 7fc45dee2180 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/04/2026 01:52:14.715 7fc45977eae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/04/2026 01:52:14.729 7fc45dee2180 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 05/04/2026 01:52:14.730 7fc45977eae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/04/2026 01:52:14.754 7fc45dee2180 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 05/04/2026 01:52:14.755 7fc45dee2180 [contrail-device-manager] [WARNING]: initializing cassandra cluster 05/04/2026 01:52:14.755 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 05/04/2026 01:52:14.756 7fc45941f220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 05/04/2026 01:52:14.757 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 05/04/2026 01:52:14.858 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 05/04/2026 01:52:14.860 7fc45941fe00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 05/04/2026 01:52:14.907 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 05/04/2026 01:52:14.910 7fc45941fd60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 05/04/2026 01:52:14.941 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 05/04/2026 01:52:15.061 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 05/04/2026 01:52:15.708 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:15.712 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 05/04/2026 01:52:15.717 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:16.214 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:16.316 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 05/04/2026 01:52:16.321 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:16.861 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:16.863 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 05/04/2026 01:52:16.866 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:17.247 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:17.336 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 05/04/2026 01:52:17.342 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:17.343 7fc45978bea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/04/2026 01:52:17.349 7fc45977ecc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 05/04/2026 01:52:17.362 7fc45977e220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/04/2026 01:52:17.872 7fc45dee2180 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 05/04/2026 01:52:18.376 7fc45dee2180 [contrail-device-manager] [WARNING]: Process 109 prepared to run in Full mode... 05/04/2026 01:52:18.376 7fc45dee2180 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 05/04/2026 01:52:18.426 7fc459846f40 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 05/04/2026 01:52:18.427 7fc459846f40 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 05/04/2026 01:52:18.836 7fc458dd3ea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 05/04/2026 01:52:18.840 7fc458dd3ea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 05/04/2026 01:52:21.864 7fc45dee2180 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 05/04/2026 01:52:21.864 7fc45dee2180 [contrail-device-manager] [ERROR]: Unable to load init data 05/04/2026 01:52:21.864 7fc45dee2180 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 05/04/2026 01:52:21.864 7fc45dee2180 [contrail-device-manager] [WARNING]: initializing cassandra cluster 05/04/2026 01:52:21.864 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 05/04/2026 01:52:21.873 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 05/04/2026 01:52:21.913 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 05/04/2026 01:52:21.920 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 05/04/2026 01:52:21.968 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:22.126 7fc45dee2180 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 05/04/2026 01:52:22.132 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:22.145 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:22.216 7fc45dee2180 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 05/04/2026 01:52:22.219 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:22.219 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 05/04/2026 01:52:22.262 7fc45dee2180 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 05/04/2026 01:52:22.275 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:22.507 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 05/04/2026 01:52:22.520 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:22.536 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:22.717 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 05/04/2026 01:52:22.723 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:22.740 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:22.970 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 05/04/2026 01:52:22.979 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:23.141 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:23.282 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 05/04/2026 01:52:23.288 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:23.373 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:23.620 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 05/04/2026 01:52:23.879 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:24.347 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:24.531 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 05/04/2026 01:52:24.545 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:24.770 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:25.380 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 05/04/2026 01:52:25.385 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:25.692 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/04/2026 01:52:26.020 7fc45dee2180 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 05/04/2026 01:52:26.029 7fc45dee2180 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/04/2026 01:52:26.180 7fc45dee2180 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//'] 05/04/2026 01:52:26.180 7fc45dee2180 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/04/2026 01:52:26.198 7fc45dee2180 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/04/2026 01:52:26.221 7fc45dee2180 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/04/2026 01:52:26.248 7fc45dee2180 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 05/04/2026 01:52:26.259 7fc458ad40e0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/04/2026 01:52:26.273 7fc458ad40e0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 05/04/2026 01:52:27.050 7fc458b71cc0 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 05/04/2026 01:52:27.050 7fc458b71cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/04/2026 01:52:27.088 7fc458b71cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/04/2026 01:52:27.120 7fc458b71cc0 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 05/04/2026 01:52:27.120 7fc458b71cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/04/2026 01:52:27.195 7fc458b71cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED