01/27/2025 02:07:07.952 7f6bea577048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/27/2025 02:07:07.952 7f6bea577048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1737943627952199 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 01/27/2025 02:07:08.188 7f6bf3056b88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 01/27/2025 02:07:08.202 7f6bf3056b88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/27/2025 02:07:08.202 7f6be9a7fd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/27/2025 02:07:08.202 7f6be9a7fd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/27/2025 02:07:08.233 7f6bf3056b88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 01/27/2025 02:07:08.233 7f6be9a7fc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/27/2025 02:07:08.414 7f6bf3056b88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/27/2025 02:07:08.416 7f6bea471f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/27/2025 02:07:08.550 7f6bf3056b88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 01/27/2025 02:07:08.557 7f6bea471f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/27/2025 02:07:08.643 7f6bf3056b88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 01/27/2025 02:07:08.645 7f6bf3056b88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 01/27/2025 02:07:08.660 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 01/27/2025 02:07:08.661 7f6bea4710f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 01/27/2025 02:07:08.662 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 01/27/2025 02:07:08.893 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 01/27/2025 02:07:08.901 7f6bea3e0e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 01/27/2025 02:07:08.979 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 01/27/2025 02:07:08.980 7f6bea4b7c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 01/27/2025 02:07:10.249 7f6bf3056b88 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/27/2025 02:07:11.285 7f6bf3056b88 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/27/2025 02:07:11.956 7f6bea3e70f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-130-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 01/27/2025 02:07:12.303 7f6bf3056b88 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/27/2025 02:07:12.423 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 01/27/2025 02:07:12.485 7f6bf3056b88 [contrail-device-manager] [ERROR]: Error while initializing db connection, retrying: ("Unable to connect to any servers using keyspace 'config_db_uuid'", ['10.0.0.23']) 01/27/2025 02:07:22.487 7f6bf3056b88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 01/27/2025 02:07:22.487 7f6bf3056b88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 01/27/2025 02:07:22.487 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 01/27/2025 02:07:22.495 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 01/27/2025 02:07:22.584 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 01/27/2025 02:07:22.717 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 01/27/2025 02:07:22.751 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 01/27/2025 02:07:22.818 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 01/27/2025 02:07:23.429 7f6bf3056b88 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))",) 01/27/2025 02:07:23.951 7f6bf3056b88 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 01/27/2025 02:07:23.951 7f6bf3056b88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 01/27/2025 02:07:24.191 7f6bea308438 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 01/27/2025 02:07:24.191 7f6bea308438 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 01/27/2025 02:07:34.001 7f6bf3056b88 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 01/27/2025 02:07:34.002 7f6bf3056b88 [contrail-device-manager] [ERROR]: Unable to load init data 01/27/2025 02:07:34.002 7f6bf3056b88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 01/27/2025 02:07:34.002 7f6bf3056b88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 01/27/2025 02:07:34.002 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 01/27/2025 02:07:34.017 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 01/27/2025 02:07:34.187 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 01/27/2025 02:07:34.236 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 01/27/2025 02:07:34.331 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/27/2025 02:07:34.576 7f6bf3056b88 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 01/27/2025 02:07:34.580 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/27/2025 02:07:34.598 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/27/2025 02:07:34.735 7f6bf3056b88 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 01/27/2025 02:07:34.744 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/27/2025 02:07:34.747 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 01/27/2025 02:07:35.019 7f6bf3056b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 01/27/2025 02:07:35.045 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/27/2025 02:07:35.315 7f6bf3056b88 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 01/27/2025 02:07:35.357 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/27/2025 02:07:35.490 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/27/2025 02:07:35.770 7f6bf3056b88 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 01/27/2025 02:07:35.795 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/27/2025 02:07:36.075 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/27/2025 02:07:36.620 7f6bf3056b88 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 01/27/2025 02:07:36.667 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/27/2025 02:07:37.235 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/27/2025 02:07:37.606 7f6bf3056b88 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 01/27/2025 02:07:37.633 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/27/2025 02:07:38.279 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/27/2025 02:07:38.594 7f6bf3056b88 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 01/27/2025 02:07:38.613 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/27/2025 02:07:39.059 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/27/2025 02:07:39.241 7f6bf3056b88 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 01/27/2025 02:07:39.246 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/27/2025 02:07:39.438 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/27/2025 02:07:39.723 7f6bf3056b88 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 01/27/2025 02:07:39.738 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/27/2025 02:07:40.405 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/27/2025 02:07:40.715 7f6bf3056b88 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 01/27/2025 02:07:40.722 7f6bf3056b88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/27/2025 02:07:40.837 7f6bf3056b88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 01/27/2025 02:07:40.837 7f6bf3056b88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/27/2025 02:07:40.863 7f6bf3056b88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 01/27/2025 02:07:40.895 7f6bf3056b88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/27/2025 02:07:40.914 7f6bf3056b88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 01/27/2025 02:07:40.917 7f6bdb4a2588 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/27/2025 02:07:40.929 7f6bdb4a2588 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED