11/10/2025 01:28:17.558 7ffb48a8eea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
11/10/2025 01:28:17.558 7ffb48a8eea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local:Config:contrail-device-manager:0  client_info = <<  status = Idle  successful_connections = 0  pid = 109  start_time = 1762738097558235  collector_name =   collector_ip =   collector_list = [ 10.0.0.33:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
11/10/2025 01:28:17.662 7ffb4d247700 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//']
11/10/2025 01:28:17.739 7ffb4d247700 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down
11/10/2025 01:28:17.745 7ffb48b8c860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33:5673,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
11/10/2025 01:28:17.745 7ffb48b8c860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33:5673,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
11/10/2025 01:28:17.763 7ffb4d247700 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7ffb48ba3190>
11/10/2025 01:28:17.763 7ffb48b8c9a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33:5673,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
11/10/2025 01:28:17.821 7ffb4d247700 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down
11/10/2025 01:28:17.821 7ffb4885f180 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33:5673,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
11/10/2025 01:28:17.846 7ffb4d247700 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7ffb48bb1d60>
11/10/2025 01:28:17.847 7ffb4885f180 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33:5673,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
11/10/2025 01:28:17.882 7ffb4d247700 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql
11/10/2025 01:28:17.883 7ffb4d247700 [contrail-device-manager] [WARNING]: initializing cassandra cluster
11/10/2025 01:28:17.883 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create
11/10/2025 01:28:17.891 7ffb4885f9a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.33:9041,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down  >>,  ]  >>  
11/10/2025 01:28:17.893 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created
11/10/2025 01:28:18.076 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected
11/10/2025 01:28:18.077 7ffb4883f860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.33:9041,  ]  status = Initializing  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.33:2181,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down  >>,  ]  >>  
11/10/2025 01:28:18.092 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS
11/10/2025 01:28:18.092 7ffb4883f7c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.33:9041,  ]  status = Initializing  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.33:2181,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down  >>,  ]  >>  
11/10/2025 01:28:19.112 7ffb4d247700 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created
11/10/2025 01:28:19.129 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready
11/10/2025 01:28:19.163 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready
11/10/2025 01:28:19.239 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:19.432 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created
11/10/2025 01:28:19.440 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:19.557 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:19.805 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000}
11/10/2025 01:28:19.814 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:20.335 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:20.343 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created
11/10/2025 01:28:20.346 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:21.012 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:21.202 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000}
11/10/2025 01:28:21.215 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:21.569 7ffb48b8c860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-269-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.33:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.33:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.33:9041,  ]  status = Initializing  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.33:2181,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down  >>,  ]  >>  
11/10/2025 01:28:21.587 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:21.601 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created
11/10/2025 01:28:21.611 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:22.058 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:22.169 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000}
11/10/2025 01:28:22.176 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:22.720 7ffb4d247700 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))")
11/10/2025 01:28:23.229 7ffb4d247700 [contrail-device-manager] [WARNING]: Process 109 prepared to run in Full mode...
11/10/2025 01:28:23.229 7ffb4d247700 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 
11/10/2025 01:28:23.318 7ffb48b395e0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port
11/10/2025 01:28:23.318 7ffb48b395e0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096
11/10/2025 01:28:26.656 7ffb4d247700 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json
11/10/2025 01:28:26.656 7ffb4d247700 [contrail-device-manager] [ERROR]: Unable to load init data
11/10/2025 01:28:26.656 7ffb4d247700 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql
11/10/2025 01:28:26.656 7ffb4d247700 [contrail-device-manager] [WARNING]: initializing cassandra cluster
11/10/2025 01:28:26.656 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create
11/10/2025 01:28:26.658 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created
11/10/2025 01:28:26.722 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected
11/10/2025 01:28:26.732 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS
11/10/2025 01:28:26.755 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:26.900 7ffb4d247700 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created
11/10/2025 01:28:26.905 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:26.911 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:27.032 7ffb4d247700 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None}
11/10/2025 01:28:27.041 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:27.043 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready
11/10/2025 01:28:27.524 7ffb4d247700 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready
11/10/2025 01:28:27.533 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:27.922 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created
11/10/2025 01:28:27.943 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:28.144 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:28.305 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000}
11/10/2025 01:28:28.309 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:28.412 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:28.612 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created
11/10/2025 01:28:28.620 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:28.842 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:28.981 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000}
11/10/2025 01:28:28.983 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:29.159 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:29.337 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created
11/10/2025 01:28:29.344 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:29.578 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:29.792 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000}
11/10/2025 01:28:29.801 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:30.014 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:30.341 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created
11/10/2025 01:28:30.349 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:30.714 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
11/10/2025 01:28:30.959 7ffb4d247700 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000}
11/10/2025 01:28:30.970 7ffb4d247700 [contrail-device-manager] [WARNING]: Zookeeper lock released.
11/10/2025 01:28:31.066 7ffb4d247700 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//']
11/10/2025 01:28:31.067 7ffb4d247700 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down
11/10/2025 01:28:31.085 7ffb4d247700 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7ffb48a9bac0>
11/10/2025 01:28:31.165 7ffb4d247700 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down
11/10/2025 01:28:31.190 7ffb4d247700 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7ffb488c0fa0>
11/10/2025 01:28:31.195 7ffb482eb360 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down
11/10/2025 01:28:31.232 7ffb482eb360 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.33:5673// at 0x7ffb488c0fa0>