03/26/2026 01:44:44.276 7f81acd2b220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/26/2026 01:44:44.277 7f81acd2b220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 107 start_time = 1774489484276621 collector_name = collector_ip = collector_list = [ 10.0.0.242:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/26/2026 01:44:44.312 7f81b15b9c80 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.242:5673//'] 03/26/2026 01:44:44.339 7f81b15b9c80 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 03/26/2026 01:44:44.340 7f81acd2bb80 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.242:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/26/2026 01:44:44.340 7f81acd2bb80 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.242:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/26/2026 01:44:44.369 7f81b15b9c80 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 03/26/2026 01:44:44.370 7f81acd2bcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.242:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/26/2026 01:44:44.409 7f81b15b9c80 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 03/26/2026 01:44:44.410 7f81acd824a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.242:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/26/2026 01:44:44.414 7f81b15b9c80 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 03/26/2026 01:44:44.414 7f81acd824a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.242:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/26/2026 01:44:44.584 7f81b15b9c80 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 03/26/2026 01:44:44.584 7f81b15b9c80 [contrail-device-manager] [WARNING]: initializing cassandra cluster 03/26/2026 01:44:44.588 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/26/2026 01:44:44.595 7f81acd82b80 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.242:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.242:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 03/26/2026 01:44:44.614 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/26/2026 01:44:44.871 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/26/2026 01:44:44.884 7f81acb617c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.242:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.242:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.242:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 03/26/2026 01:44:44.895 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 03/26/2026 01:44:44.896 7f81acb61720 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.242:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.242:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.242:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 03/26/2026 01:44:45.913 7f81b15b9c80 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 03/26/2026 01:44:45.922 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 03/26/2026 01:44:45.943 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 03/26/2026 01:44:46.496 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:46.503 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 03/26/2026 01:44:46.513 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:47.279 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:47.498 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 03/26/2026 01:44:47.525 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:47.899 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:47.900 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 03/26/2026 01:44:47.902 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:48.238 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:48.277 7f81acbb5a40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-385-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.242:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.242:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.242:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.242:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 03/26/2026 01:44:48.318 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 03/26/2026 01:44:48.323 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:48.848 7f81b15b9c80 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 03/26/2026 01:44:49.351 7f81b15b9c80 [contrail-device-manager] [WARNING]: Process 107 prepared to run in Full mode... 03/26/2026 01:44:49.352 7f81b15b9c80 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 03/26/2026 01:44:49.403 7f81acdc9ae0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 03/26/2026 01:44:49.403 7f81acdc9ae0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 03/26/2026 01:44:52.668 7f81b15b9c80 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 03/26/2026 01:44:52.669 7f81b15b9c80 [contrail-device-manager] [ERROR]: Unable to load init data 03/26/2026 01:44:52.672 7f81b15b9c80 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 03/26/2026 01:44:52.672 7f81b15b9c80 [contrail-device-manager] [WARNING]: initializing cassandra cluster 03/26/2026 01:44:52.672 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/26/2026 01:44:52.681 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/26/2026 01:44:52.774 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/26/2026 01:44:52.789 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 03/26/2026 01:44:52.827 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:52.936 7f81b15b9c80 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 03/26/2026 01:44:52.941 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:52.951 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:53.013 7f81b15b9c80 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 03/26/2026 01:44:53.016 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:53.016 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 03/26/2026 01:44:53.106 7f81b15b9c80 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 03/26/2026 01:44:53.118 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:53.293 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 03/26/2026 01:44:53.328 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:53.467 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:53.578 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 03/26/2026 01:44:53.585 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:53.738 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:53.989 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 03/26/2026 01:44:54.000 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:54.184 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:54.317 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 03/26/2026 01:44:54.321 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:54.487 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:54.646 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 03/26/2026 01:44:54.651 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:54.809 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:54.962 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 03/26/2026 01:44:54.967 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:55.131 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:55.337 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 03/26/2026 01:44:55.346 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:55.569 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/26/2026 01:44:55.676 7f81b15b9c80 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 03/26/2026 01:44:55.681 7f81b15b9c80 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/26/2026 01:44:55.718 7f81b15b9c80 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.242:5673//'] 03/26/2026 01:44:55.719 7f81b15b9c80 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 03/26/2026 01:44:55.731 7f81b15b9c80 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 03/26/2026 01:44:55.785 7f81b15b9c80 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 03/26/2026 01:44:55.796 7f81b15b9c80 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 03/26/2026 01:44:55.806 7f81ac590d60 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 03/26/2026 01:44:55.830 7f81ac590d60 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED