04/17/2025 01:59:33.583 7f45739d3588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/17/2025 01:59:33.584 7f45739d3588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1744855173550705 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 04/17/2025 01:59:34.301 7f457c4dd208 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 04/17/2025 01:59:34.336 7f457c4dd208 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 04/17/2025 01:59:34.339 7f45736b54e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/17/2025 01:59:34.342 7f45736b54e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/17/2025 01:59:34.362 7f457c4dd208 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/17/2025 01:59:34.373 7f45736c8c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/17/2025 01:59:34.421 7f457c4dd208 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/17/2025 01:59:34.432 7f45736b5e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/17/2025 01:59:34.442 7f457c4dd208 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/17/2025 01:59:34.454 7f45736b5eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/17/2025 01:59:34.968 7f457c4dd208 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 04/17/2025 01:59:34.969 7f457c4dd208 [contrail-device-manager] [WARNING]: initializing cassandra cluster 04/17/2025 01:59:34.969 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 04/17/2025 01:59:34.970 7f45736d1978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 04/17/2025 01:59:34.984 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 04/17/2025 01:59:35.362 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 04/17/2025 01:59:35.425 7f45738fda20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 04/17/2025 01:59:35.834 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 04/17/2025 01:59:35.834 7f4573956828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 04/17/2025 01:59:37.158 7f457c4dd208 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 04/17/2025 01:59:37.555 7f457394b198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-160-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 04/17/2025 01:59:38.498 7f457c4dd208 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 04/17/2025 01:59:39.903 7f457c4dd208 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 04/17/2025 01:59:41.181 7f457c4dd208 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 04/17/2025 01:59:42.378 7f457c4dd208 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 04/17/2025 01:59:43.644 7f457c4dd208 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 04/17/2025 01:59:44.675 7f457c4dd208 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 04/17/2025 01:59:45.840 7f457c4dd208 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 04/17/2025 01:59:45.928 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 04/17/2025 01:59:46.092 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 04/17/2025 01:59:47.202 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 01:59:47.214 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 04/17/2025 01:59:47.262 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 01:59:48.292 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 01:59:48.463 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 04/17/2025 01:59:48.484 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 01:59:49.627 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 01:59:49.639 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 04/17/2025 01:59:49.651 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 01:59:50.436 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 01:59:50.708 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 04/17/2025 01:59:50.729 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 01:59:51.818 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 01:59:51.823 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 04/17/2025 01:59:51.841 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 01:59:52.460 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 01:59:52.669 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 04/17/2025 01:59:52.679 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 01:59:53.241 7f457c4dd208 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))",) 04/17/2025 01:59:53.758 7f457c4dd208 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 04/17/2025 01:59:53.790 7f457c4dd208 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 04/17/2025 01:59:56.134 7f45737bb630 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 04/17/2025 01:59:56.135 7f45737bb630 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 04/17/2025 02:01:19.440 7f457c4dd208 [contrail-device-manager] [WARNING]: NoIdError: default-domain:admin:master-LR 04/17/2025 02:01:19.581 7f457c4dd208 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_template 04/17/2025 02:01:19.627 7f457c4dd208 [contrail-device-manager] [WARNING]: NoIdError: show_config_interfaces_template 04/17/2025 02:01:19.673 7f457c4dd208 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_by_names_template 04/17/2025 02:01:19.674 7f457c4dd208 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 04/17/2025 02:01:19.674 7f457c4dd208 [contrail-device-manager] [WARNING]: initializing cassandra cluster 04/17/2025 02:01:19.674 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 04/17/2025 02:01:19.676 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 04/17/2025 02:01:19.703 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 04/17/2025 02:01:19.710 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 04/17/2025 02:01:19.720 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 02:01:19.788 7f457c4dd208 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 04/17/2025 02:01:19.792 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 02:01:19.802 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 02:01:19.848 7f457c4dd208 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 04/17/2025 02:01:19.855 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 02:01:19.855 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 04/17/2025 02:01:19.881 7f457c4dd208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 04/17/2025 02:01:19.886 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 02:01:20.035 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 04/17/2025 02:01:20.043 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 02:01:20.051 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 02:01:20.162 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 04/17/2025 02:01:20.167 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 02:01:20.174 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 02:01:20.294 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 04/17/2025 02:01:20.299 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 02:01:20.306 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 02:01:20.383 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 04/17/2025 02:01:20.387 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 02:01:20.391 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 02:01:20.520 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 04/17/2025 02:01:20.530 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 02:01:20.542 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 02:01:20.610 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 04/17/2025 02:01:20.613 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 02:01:20.619 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 02:01:20.767 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 04/17/2025 02:01:20.770 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 02:01:20.778 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/17/2025 02:01:20.872 7f457c4dd208 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 04/17/2025 02:01:20.876 7f457c4dd208 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/17/2025 02:01:20.916 7f457c4dd208 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 04/17/2025 02:01:20.917 7f457c4dd208 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 04/17/2025 02:01:20.936 7f457c4dd208 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/17/2025 02:01:20.948 7f457c4dd208 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/17/2025 02:01:20.959 7f457c4dd208 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/17/2025 02:01:20.964 7f45732ef780 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/17/2025 02:01:20.992 7f45732ef780 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED