11/13/2025 01:40:11.283 7f2dd0927ea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/13/2025 01:40:11.283 7f2dd0927ea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1762998011283039 collector_name = collector_ip = collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/13/2025 01:40:11.305 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//'] 11/13/2025 01:40:11.312 7f2dd4f67e80 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/13/2025 01:40:11.313 7f2dd0a2f860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/13/2025 01:40:11.313 7f2dd0a2f860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/13/2025 01:40:11.328 7f2dd4f67e80 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/13/2025 01:40:11.328 7f2dd0a2f9a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/13/2025 01:40:11.361 7f2dd4f67e80 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/13/2025 01:40:11.362 7f2dd0702180 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/13/2025 01:40:11.372 7f2dd4f67e80 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/13/2025 01:40:11.372 7f2dd0702180 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/13/2025 01:40:11.417 7f2dd4f67e80 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/13/2025 01:40:11.417 7f2dd4f67e80 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/13/2025 01:40:11.417 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/13/2025 01:40:11.421 7f2dd07029a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/13/2025 01:40:11.422 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/13/2025 01:40:11.700 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/13/2025 01:40:11.720 7f2dd0a2f5e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/13/2025 01:40:11.753 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/13/2025 01:40:11.754 7f2dd0a2f4a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/13/2025 01:40:11.782 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/13/2025 01:40:11.854 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/13/2025 01:40:11.855 7f2dd06e3d60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/13/2025 01:40:11.868 7f2dd0702860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/13/2025 01:40:11.923 7f2dd0702860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/13/2025 01:40:12.447 7f2dd4f67e80 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 11/13/2025 01:40:12.949 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/13/2025 01:40:12.950 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/13/2025 01:40:13.089 7f2dd09c5040 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/13/2025 01:40:13.090 7f2dd09c5040 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/13/2025 01:40:13.707 7f2dcbe88d60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.33,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/13/2025 01:40:13.753 7f2dcbe88f40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.33,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/13/2025 01:40:15.284 7f2dcbfa9040 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-270-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.33,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/13/2025 01:40:16.831 7f2dd4f67e80 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 11/13/2025 01:40:16.831 7f2dd4f67e80 [contrail-device-manager] [ERROR]: Unable to load init data 11/13/2025 01:40:16.831 7f2dd4f67e80 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/13/2025 01:40:16.831 7f2dd4f67e80 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/13/2025 01:40:16.831 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/13/2025 01:40:16.839 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/13/2025 01:40:17.019 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/13/2025 01:40:17.043 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/13/2025 01:40:17.116 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/13/2025 01:40:17.286 7f2dd4f67e80 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 11/13/2025 01:40:17.296 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/13/2025 01:40:17.521 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/13/2025 01:40:17.671 7f2dd4f67e80 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 11/13/2025 01:40:17.694 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/13/2025 01:40:17.694 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/13/2025 01:40:18.017 7f2dd4f67e80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/13/2025 01:40:18.041 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/13/2025 01:40:18.221 7f2dd4f67e80 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 11/13/2025 01:40:18.232 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/13/2025 01:40:18.750 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/13/2025 01:40:18.939 7f2dd4f67e80 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 11/13/2025 01:40:18.951 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/13/2025 01:40:19.194 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/13/2025 01:40:19.718 7f2dd4f67e80 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 11/13/2025 01:40:19.733 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/13/2025 01:40:20.025 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/13/2025 01:40:20.166 7f2dd4f67e80 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 11/13/2025 01:40:20.178 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/13/2025 01:40:20.397 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/13/2025 01:40:20.656 7f2dd4f67e80 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 11/13/2025 01:40:20.663 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/13/2025 01:40:21.062 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/13/2025 01:40:21.308 7f2dd4f67e80 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 11/13/2025 01:40:21.317 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/13/2025 01:40:21.609 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/13/2025 01:40:21.828 7f2dd4f67e80 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 11/13/2025 01:40:21.844 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/13/2025 01:40:22.646 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/13/2025 01:40:22.850 7f2dd4f67e80 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 11/13/2025 01:40:22.866 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/13/2025 01:40:22.906 7f2dd4f67e80 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//'] 11/13/2025 01:40:22.906 7f2dd4f67e80 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/13/2025 01:40:22.920 7f2dd4f67e80 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/13/2025 01:40:22.952 7f2dd4f67e80 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/13/2025 01:40:22.960 7f2dd4f67e80 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/13/2025 01:40:22.962 7f2dcb9dbea0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/13/2025 01:40:22.972 7f2dcb9dbea0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED