08/18/2025 01:40:37.598 7face7173438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/18/2025 01:40:37.599 7face7173438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 111 start_time = 1755481237598407 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 08/18/2025 01:40:37.642 7facef27ea08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 08/18/2025 01:40:37.651 7facef27ea08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 08/18/2025 01:40:37.652 7face65c3e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/18/2025 01:40:37.652 7face65c3e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/18/2025 01:40:37.703 7facef27ea08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/18/2025 01:40:37.704 7face65e66d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/18/2025 01:40:37.866 7facef27ea08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 08/18/2025 01:40:37.867 7face65f1630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/18/2025 01:40:37.901 7facef27ea08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/18/2025 01:40:37.902 7face65f1630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/18/2025 01:40:38.216 7facef27ea08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 08/18/2025 01:40:38.217 7facef27ea08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 08/18/2025 01:40:38.219 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 08/18/2025 01:40:38.220 7face65f1390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 08/18/2025 01:40:38.221 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 08/18/2025 01:40:38.649 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 08/18/2025 01:40:38.651 7face66bd390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 08/18/2025 01:40:38.946 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 08/18/2025 01:40:38.946 7face66b7198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 08/18/2025 01:40:40.414 7facef27ea08 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 08/18/2025 01:40:41.612 7face6605ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/18/2025 01:40:41.698 7facef27ea08 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 08/18/2025 01:40:42.766 7facef27ea08 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 08/18/2025 01:40:43.867 7facef27ea08 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 08/18/2025 01:40:44.882 7facef27ea08 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 08/18/2025 01:40:45.906 7facef27ea08 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 08/18/2025 01:40:46.973 7facef27ea08 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 08/18/2025 01:40:47.023 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 08/18/2025 01:40:47.176 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 08/18/2025 01:40:47.926 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:40:47.933 7facef27ea08 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 08/18/2025 01:40:47.958 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:40:49.067 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:40:49.990 7facef27ea08 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 08/18/2025 01:40:50.002 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:40:50.359 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:40:50.364 7facef27ea08 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 08/18/2025 01:40:50.369 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:40:50.567 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:40:50.661 7facef27ea08 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 08/18/2025 01:40:50.665 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:40:51.056 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:40:51.060 7facef27ea08 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 08/18/2025 01:40:51.064 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:40:51.344 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:40:51.472 7facef27ea08 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 08/18/2025 01:40:51.482 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:40:52.021 7facef27ea08 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))",) 08/18/2025 01:40:52.525 7facef27ea08 [contrail-device-manager] [WARNING]: Process 111 prepared to run in Full mode... 08/18/2025 01:40:52.525 7facef27ea08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 08/18/2025 01:40:53.462 7face64e14e0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 08/18/2025 01:40:53.462 7face64e14e0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 08/18/2025 01:42:05.648 7facef27ea08 [contrail-device-manager] [WARNING]: NoIdError: default-domain:admin:master-LR 08/18/2025 01:42:05.785 7facef27ea08 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_template 08/18/2025 01:42:05.830 7facef27ea08 [contrail-device-manager] [WARNING]: NoIdError: show_config_interfaces_template 08/18/2025 01:42:05.875 7facef27ea08 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_by_names_template 08/18/2025 01:42:05.875 7facef27ea08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 08/18/2025 01:42:05.876 7facef27ea08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 08/18/2025 01:42:05.876 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 08/18/2025 01:42:05.877 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 08/18/2025 01:42:05.909 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 08/18/2025 01:42:05.922 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 08/18/2025 01:42:05.951 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:42:06.057 7facef27ea08 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 08/18/2025 01:42:06.066 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:42:06.080 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:42:06.113 7facef27ea08 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 08/18/2025 01:42:06.116 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:42:06.116 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 08/18/2025 01:42:06.135 7facef27ea08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 08/18/2025 01:42:06.140 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:42:06.223 7facef27ea08 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 08/18/2025 01:42:06.225 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:42:06.232 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:42:06.282 7facef27ea08 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 08/18/2025 01:42:06.290 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:42:06.306 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:42:06.378 7facef27ea08 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 08/18/2025 01:42:06.381 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:42:06.389 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:42:06.433 7facef27ea08 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 08/18/2025 01:42:06.438 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:42:06.443 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:42:06.514 7facef27ea08 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 08/18/2025 01:42:06.520 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:42:06.526 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:42:06.569 7facef27ea08 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 08/18/2025 01:42:06.571 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:42:06.575 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:42:06.668 7facef27ea08 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 08/18/2025 01:42:06.671 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:42:06.675 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/18/2025 01:42:06.724 7facef27ea08 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 08/18/2025 01:42:06.726 7facef27ea08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/18/2025 01:42:06.745 7facef27ea08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 08/18/2025 01:42:06.746 7facef27ea08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 08/18/2025 01:42:06.751 7facef27ea08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/18/2025 01:42:06.758 7facef27ea08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 08/18/2025 01:42:06.762 7facef27ea08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/18/2025 01:42:06.763 7face5fd6588 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 08/18/2025 01:42:06.774 7face5fd6588 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/18/2025 01:43:13.168 7face5fd6cc0 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 08/18/2025 01:43:13.168 7face5fd6cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 08/18/2025 01:43:13.185 7face5fd6cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/18/2025 01:43:13.332 7face5fd6cc0 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 08/18/2025 01:43:13.332 7face5fd6cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 08/18/2025 01:43:13.374 7face5fd6cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/18/2025 01:43:14.129 7face5fd6cc0 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 08/18/2025 01:43:14.129 7face5fd6cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 08/18/2025 01:43:14.154 7face5fd6cc0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED