04/24/2025 01:33:55.532 7fcf9fde06d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/24/2025 01:33:55.532 7fcf9fde06d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 111 start_time = 1745458435532435 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 04/24/2025 01:33:55.592 7fcfac922f88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 04/24/2025 01:33:55.603 7fcfac922f88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 04/24/2025 01:33:55.603 7fcf9fabc4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/24/2025 01:33:55.603 7fcf9fabc4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/24/2025 01:33:55.613 7fcfac922f88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/24/2025 01:33:55.614 7fcf9fad7c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/24/2025 01:33:55.630 7fcfac922f88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/24/2025 01:33:55.631 7fcf9fabce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/24/2025 01:33:55.636 7fcfac922f88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/24/2025 01:33:55.637 7fcfafb16eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/24/2025 01:33:55.992 7fcfac922f88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 04/24/2025 01:33:55.992 7fcfac922f88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 04/24/2025 01:33:55.992 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 04/24/2025 01:33:55.993 7fcf9fabc8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 04/24/2025 01:33:55.995 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 04/24/2025 01:33:56.034 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 04/24/2025 01:33:56.056 7fcf9fd67438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 04/24/2025 01:33:56.389 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 04/24/2025 01:33:56.389 7fcf9fd5b240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 04/24/2025 01:33:56.398 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 04/24/2025 01:33:56.418 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 04/24/2025 01:33:56.741 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:33:57.018 7fcfac922f88 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 04/24/2025 01:33:57.030 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:33:57.656 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:33:57.952 7fcfac922f88 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 01:33:57.962 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:33:57.971 7fcf9fd26198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/24/2025 01:33:58.102 7fcf9fbf9438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 04/24/2025 01:33:58.175 7fcf9fbfd2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/24/2025 01:33:58.683 7fcfac922f88 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))",) 04/24/2025 01:33:59.192 7fcfac922f88 [contrail-device-manager] [WARNING]: Process 111 prepared to run in Full mode... 04/24/2025 01:33:59.192 7fcfac922f88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 04/24/2025 01:33:59.653 7fcf9fc03d68 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 04/24/2025 01:33:59.654 7fcf9fc03d68 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 04/24/2025 01:33:59.654 7fcf9f8b76d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 04/24/2025 01:33:59.658 7fcf9f8b76d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-162-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 04/24/2025 01:35:15.084 7fcfac922f88 [contrail-device-manager] [WARNING]: NoIdError: default-domain:admin:master-LR 04/24/2025 01:35:15.219 7fcfac922f88 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_template 04/24/2025 01:35:15.264 7fcfac922f88 [contrail-device-manager] [WARNING]: NoIdError: show_config_interfaces_template 04/24/2025 01:35:15.309 7fcfac922f88 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_by_names_template 04/24/2025 01:35:15.309 7fcfac922f88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 04/24/2025 01:35:15.309 7fcfac922f88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 04/24/2025 01:35:15.310 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 04/24/2025 01:35:15.311 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 04/24/2025 01:35:15.329 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 04/24/2025 01:35:15.335 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 04/24/2025 01:35:15.344 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:35:15.380 7fcfac922f88 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 04/24/2025 01:35:15.382 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:35:15.386 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:35:15.417 7fcfac922f88 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 04/24/2025 01:35:15.421 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:35:15.421 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 04/24/2025 01:35:15.442 7fcfac922f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 04/24/2025 01:35:15.451 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:35:15.541 7fcfac922f88 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 04/24/2025 01:35:15.543 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:35:15.548 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:35:15.595 7fcfac922f88 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 01:35:15.601 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:35:15.607 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:35:15.688 7fcfac922f88 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 04/24/2025 01:35:15.693 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:35:15.709 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:35:15.775 7fcfac922f88 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 01:35:15.781 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:35:15.787 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:35:15.866 7fcfac922f88 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 04/24/2025 01:35:15.871 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:35:15.875 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:35:15.928 7fcfac922f88 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 01:35:15.930 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:35:15.935 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:35:16.029 7fcfac922f88 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 04/24/2025 01:35:16.032 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:35:16.036 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 01:35:16.090 7fcfac922f88 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 01:35:16.092 7fcfac922f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 01:35:16.109 7fcfac922f88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 04/24/2025 01:35:16.109 7fcfac922f88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 04/24/2025 01:35:16.114 7fcfac922f88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/24/2025 01:35:16.119 7fcfac922f88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/24/2025 01:35:16.123 7fcfac922f88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/24/2025 01:35:16.124 7fcf9f6ea390 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/24/2025 01:35:16.132 7fcf9f6ea390 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED