10/09/2025 01:35:40.849 7f6645afc400 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
10/09/2025 01:35:40.849 7f6645afc400 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local:Config:contrail-device-manager:0  client_info = <<  status = Idle  successful_connections = 0  pid = 107  start_time = 1759973740849468  collector_name =   collector_ip =   collector_list = [ 10.0.0.49:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
10/09/2025 01:35:40.870 7f664a90d600 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//']
10/09/2025 01:35:40.877 7f664a90d600 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down
10/09/2025 01:35:40.877 7f6645afccc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.49:5673,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
10/09/2025 01:35:40.878 7f6645afccc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.49:5673,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
10/09/2025 01:35:40.882 7f664a90d600 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.49:5673// at 0x7f6646357850>
10/09/2025 01:35:40.882 7f6645afce00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.49:5673,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
10/09/2025 01:35:40.892 7f664a90d600 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down
10/09/2025 01:35:40.892 7f6645a765e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.49:5673,  ]  status = Down  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down  >>,  ]  >>  
10/09/2025 01:35:40.896 7f664a90d600 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.49:5673// at 0x7f6645b27be0>
10/09/2025 01:35:40.897 7f6645a765e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.49:5673,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
10/09/2025 01:35:40.910 7f664a90d600 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql
10/09/2025 01:35:40.910 7f664a90d600 [contrail-device-manager] [WARNING]: initializing cassandra cluster
10/09/2025 01:35:40.910 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create
10/09/2025 01:35:40.911 7f6645a76e00 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.49:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.49:9041,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down  >>,  ]  >>  
10/09/2025 01:35:40.913 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created
10/09/2025 01:35:40.998 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected
10/09/2025 01:35:40.999 7f6645965720 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.49:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.49:9041,  ]  status = Initializing  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.49:2181,  ]  status = Initializing  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down  >>,  ]  >>  
10/09/2025 01:35:41.004 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS
10/09/2025 01:35:41.005 7f6645965680 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.49:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.49:9041,  ]  status = Initializing  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.49:2181,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down  >>,  ]  >>  
10/09/2025 01:35:42.013 7f664a90d600 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created
10/09/2025 01:35:43.026 7f664a90d600 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created
10/09/2025 01:35:43.034 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready
10/09/2025 01:35:43.046 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready
10/09/2025 01:35:43.256 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:44.557 7f664a90d600 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created
10/09/2025 01:35:44.586 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:44.853 7f6645754ae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local  process_status = [ <<  module_id = contrail-device-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.49:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.0.0.49:5673,  ]  status = Up  description =   >>, <<  type = Database  name = Cassandra  server_addrs = [ 10.0.0.49:9041,  ]  status = Initializing  description =   >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.0.0.49:2181,  ]  status = Up  description =   >>,  ]  description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down  >>,  ]  >>  
10/09/2025 01:35:45.379 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:45.964 7f664a90d600 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000}
10/09/2025 01:35:46.002 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:46.644 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:46.647 7f664a90d600 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created
10/09/2025 01:35:46.649 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:47.041 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:47.188 7f664a90d600 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000}
10/09/2025 01:35:47.194 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:47.594 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:47.604 7f664a90d600 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created
10/09/2025 01:35:47.613 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:47.878 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:48.791 7f664a90d600 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000}
10/09/2025 01:35:48.798 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:49.470 7f664a90d600 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))")
10/09/2025 01:35:49.972 7f664a90d600 [contrail-device-manager] [WARNING]: Process 107 prepared to run in Full mode...
10/09/2025 01:35:49.972 7f664a90d600 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 
10/09/2025 01:35:50.022 7f66459c9e00 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port
10/09/2025 01:35:50.022 7f66459c9e00 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096
10/09/2025 01:35:56.600 7f664a90d600 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json
10/09/2025 01:35:56.600 7f664a90d600 [contrail-device-manager] [ERROR]: Unable to load init data
10/09/2025 01:35:56.600 7f664a90d600 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql
10/09/2025 01:35:56.600 7f664a90d600 [contrail-device-manager] [WARNING]: initializing cassandra cluster
10/09/2025 01:35:56.600 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create
10/09/2025 01:35:56.603 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created
10/09/2025 01:35:56.767 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected
10/09/2025 01:35:56.808 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS
10/09/2025 01:35:56.967 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:57.091 7f664a90d600 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created
10/09/2025 01:35:57.102 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:57.244 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:57.337 7f664a90d600 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None}
10/09/2025 01:35:57.341 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:57.341 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready
10/09/2025 01:35:57.514 7f664a90d600 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready
10/09/2025 01:35:57.532 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:57.641 7f664a90d600 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created
10/09/2025 01:35:57.648 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:57.864 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:58.047 7f664a90d600 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000}
10/09/2025 01:35:58.068 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:58.345 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:58.512 7f664a90d600 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created
10/09/2025 01:35:58.524 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:58.680 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:58.844 7f664a90d600 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000}
10/09/2025 01:35:58.849 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:59.131 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:59.313 7f664a90d600 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created
10/09/2025 01:35:59.322 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:59.591 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:59.698 7f664a90d600 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000}
10/09/2025 01:35:59.724 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:35:59.776 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:35:59.973 7f664a90d600 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created
10/09/2025 01:35:59.987 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:36:00.011 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change...
10/09/2025 01:36:00.155 7f664a90d600 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000}
10/09/2025 01:36:00.163 7f664a90d600 [contrail-device-manager] [WARNING]: Zookeeper lock released.
10/09/2025 01:36:00.196 7f664a90d600 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//']
10/09/2025 01:36:00.196 7f664a90d600 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down
10/09/2025 01:36:00.213 7f664a90d600 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.49:5673// at 0x7f6645b36d90>
10/09/2025 01:36:00.231 7f664a90d600 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down
10/09/2025 01:36:00.311 7f664a90d600 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.49:5673// at 0x7f6645794760>
10/09/2025 01:36:00.333 7f6644dd5d60 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down
10/09/2025 01:36:00.371 7f6644dd5d60 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <Connection: amqp://guest:**@10.0.0.49:5673// at 0x7f6645794760>