11/17/2025 01:32:22.144 7fa71a0e8ea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/17/2025 01:32:22.145 7fa71a0e8ea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 105 start_time = 1763343142144659 collector_name = collector_ip = collector_list = [ 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/17/2025 01:32:22.175 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//'] 11/17/2025 01:32:22.184 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/17/2025 01:32:22.185 7fa71a388860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/17/2025 01:32:22.185 7fa71a388860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/17/2025 01:32:22.193 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/17/2025 01:32:22.193 7fa71a3889a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/17/2025 01:32:22.206 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/17/2025 01:32:22.207 7fa71a052180 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/17/2025 01:32:22.211 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/17/2025 01:32:22.212 7fa71a052180 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/17/2025 01:32:22.227 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/17/2025 01:32:22.227 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/17/2025 01:32:22.228 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/17/2025 01:32:22.229 7fa71a0529a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/17/2025 01:32:22.230 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/17/2025 01:32:22.352 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/17/2025 01:32:22.353 7fa71a032860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/17/2025 01:32:22.362 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/17/2025 01:32:22.363 7fa71a0327c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/17/2025 01:32:23.372 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 11/17/2025 01:32:23.385 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/17/2025 01:32:23.464 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/17/2025 01:32:23.710 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:23.936 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 11/17/2025 01:32:23.947 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:24.140 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:24.260 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 11/17/2025 01:32:24.299 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:26.145 7fa71a34a360 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-271-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 11/17/2025 01:32:26.178 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:26.182 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 11/17/2025 01:32:26.274 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:27.057 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:27.306 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 11/17/2025 01:32:27.374 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:27.913 7fa71e8b7cc0 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 11/17/2025 01:32:28.420 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Process 105 prepared to run in Full mode... 11/17/2025 01:32:28.420 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/17/2025 01:32:28.482 7fa71a22e5e0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/17/2025 01:32:28.482 7fa71a22e5e0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/17/2025 01:32:31.766 7fa71e8b7cc0 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 11/17/2025 01:32:31.766 7fa71e8b7cc0 [contrail-device-manager] [ERROR]: Unable to load init data 11/17/2025 01:32:31.767 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/17/2025 01:32:31.767 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/17/2025 01:32:31.767 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/17/2025 01:32:31.769 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/17/2025 01:32:31.802 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/17/2025 01:32:31.809 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/17/2025 01:32:31.830 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:31.878 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 11/17/2025 01:32:31.882 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:31.935 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:31.982 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 11/17/2025 01:32:31.987 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:31.987 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/17/2025 01:32:32.013 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/17/2025 01:32:32.111 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:32.239 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 11/17/2025 01:32:32.242 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:32.381 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:32.483 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 11/17/2025 01:32:32.486 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:32.820 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:32.983 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 11/17/2025 01:32:33.000 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:33.119 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:33.328 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 11/17/2025 01:32:33.335 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:33.807 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:34.108 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 11/17/2025 01:32:34.119 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:34.411 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:34.585 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 11/17/2025 01:32:34.602 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:34.854 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:35.125 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 11/17/2025 01:32:35.131 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:35.401 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/17/2025 01:32:35.576 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 11/17/2025 01:32:35.594 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/17/2025 01:32:35.646 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//'] 11/17/2025 01:32:35.646 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/17/2025 01:32:35.661 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/17/2025 01:32:35.691 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/17/2025 01:32:35.745 7fa71e8b7cc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/17/2025 01:32:35.760 7fa7192b5680 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/17/2025 01:32:35.849 7fa7192b5680 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/17/2025 01:32:36.391 7fa7197387c0 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 11/17/2025 01:32:36.391 7fa7197387c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/17/2025 01:32:36.406 7fa7197387c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/17/2025 01:32:36.413 7fa7197387c0 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 11/17/2025 01:32:36.413 7fa7197387c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/17/2025 01:32:36.466 7fa7197387c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED