2026-02-02 01:53:50,288 - DEBUG - [10.0.0.129]: Running cmd : hostname 2026-02-02 01:53:50,558 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1829-1 2026-02-02 01:53:50,558 - DEBUG - [10.0.0.129]: Running cmd : hostname -f 2026-02-02 01:53:50,591 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1829-1 2026-02-02 01:53:50,591 - DEBUG - [10.0.0.129]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-02 01:53:50,666 - DEBUG - Output : NAMES contrail_test_7WJsiR26Z webui-web-1 webui-job-1 contrail-kubernetes-master-kubemanager-1 control-named-1 control-dns-1 control-nodemgr-1 control-control-1 config_database-cassandra-1 config_database-rabbitmq-1 config_database-zookeeper-1 config_database-nodemgr-1 config_api-devicemgr-1 config_api-schema-1 config_api-svcmonitor-1 config_api-dnsmasq-1 config_api-api-1 config_api-nodemgr-1 redis-redis-1 analytics_snmp-snmp-collector-1 analytics_snmp-nodemgr-1 analytics_snmp-topology-1 analytics_alarm-alarm-gen-1 analytics_alarm-nodemgr-1 analytics_alarm-kafka-1 analytics-nodemgr-1 analytics-collector-1 analytics-api-1 analytics_database-cassandra-1 analytics_database-nodemgr-1 analytics_database-query-engine-1 2026-02-02 01:53:50,666 - DEBUG - [10.0.0.129]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-02-02 01:53:50,700 - DEBUG - Output : 127.0.0.1/8 10.0.0.129/24 252.129.0.1/8 10.108.221.1/24 172.17.0.1/16 2026-02-02 01:53:50,700 - DEBUG - [10.0.0.129]: Running cmd : getent hosts 10.0.0.129 | head -n 1 | awk '{print $2}' 2026-02-02 01:53:50,787 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1829-1 2026-02-02 01:53:50,854 - DEBUG - [10.0.0.242]: Running cmd : hostname 2026-02-02 01:53:51,082 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1829-1 2026-02-02 01:53:51,082 - DEBUG - [10.0.0.242]: Running cmd : hostname -f 2026-02-02 01:53:51,105 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1829-1 2026-02-02 01:53:51,105 - DEBUG - [10.0.0.242]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-02 01:53:51,192 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2026-02-02 01:53:51,192 - DEBUG - [10.0.0.242]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-02-02 01:53:51,227 - DEBUG - Output : 10.0.0.242/24 2026-02-02 01:53:51,227 - DEBUG - [10.0.0.242]: Running cmd : getent hosts 10.0.0.242 | head -n 1 | awk '{print $2}' 2026-02-02 01:53:51,292 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1829-1 2026-02-02 01:53:51,386 - INFO - Using existing project ['default-domain', 'k8s-default'](294bb8df-e2a6-4dfe-84f5-a89c55076f65) 2026-02-02 01:53:51,473 - INFO - Using existing project ['default-domain', 'k8s-default'](294bb8df-e2a6-4dfe-84f5-a89c55076f65) 2026-02-02 01:53:51,589 - INFO - ================================================================================ 2026-02-02 01:53:51,589 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-02-02 01:53:51,590 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-02-02 01:53:51,828 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-02 01:53:51,835 - DEBUG - Nothing to compare xmpp stats {'10.0.0.242': {'10.0.0.129': '0'}} with 2026-02-02 01:53:51,835 - INFO - Initial checks done. Running the testcase now 2026-02-02 01:53:51,835 - INFO - 2026-02-02 01:53:51,839 - INFO - Namespace default already exists 2026-02-02 01:53:51,839 - DEBUG - [10.0.0.129]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-02 01:53:51,955 - DEBUG - Output : cluster_project={} 2026-02-02 01:53:52,055 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-02-02 01:53:52,056 - DEBUG - Project uuid in Contrail is 294bb8df-e2a6-4dfe-84f5-a89c55076f65 2026-02-02 01:53:52,056 - DEBUG - [10.0.0.129]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-02-02 01:53:52,199 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-02-02 01:53:52,199 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_NamespaceDatabaseList?namespace_uuid=3664873c-1b2d-47bc-9089-7a82354a7702 2026-02-02 01:53:52,203 - INFO - Namespace default with uuid 3664873c-1b2d-47bc-9089-7a82354a7702 found in kube manager 2026-02-02 01:53:52,204 - INFO - Namespace default verification passed 2026-02-02 01:53:52,207 - DEBUG - Service ctest-nginx-svc-07227689 not present 2026-02-02 01:53:52,207 - INFO - Creating service ctest-nginx-svc-07227689 2026-02-02 01:53:52,234 - INFO - Created Service ctest-nginx-svc-07227689 2026-02-02 01:53:52,243 - DEBUG - Pod ctest-nginx-pod-29093811 not present 2026-02-02 01:53:52,244 - INFO - Creating Pod ctest-nginx-pod-29093811 2026-02-02 01:53:52,306 - DEBUG - Pod : ctest-nginx-pod-29093811 UUID is cf7bd11f-0237-4ff6-ab99-710d11cef5d8 2026-02-02 01:53:52,312 - DEBUG - Pod ctest-nginx-pod-72027969 not present 2026-02-02 01:53:52,321 - INFO - Creating Pod ctest-nginx-pod-72027969 2026-02-02 01:53:52,355 - DEBUG - Pod : ctest-nginx-pod-72027969 UUID is 09cfce5a-38ec-48be-a689-20d1b04f649c 2026-02-02 01:53:52,559 - INFO - Created VN __public__, UUID :24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:52,734 - DEBUG - Requesting: http://10.0.0.129:8082/domains 2026-02-02 01:53:52,741 - DEBUG - Requesting: http://10.0.0.129:8082/domain/7b9fb27b-8f56-456e-a3ed-374aed0f8bb9 2026-02-02 01:53:52,756 - DEBUG - Requesting: http://10.0.0.129:8082/project/294bb8df-e2a6-4dfe-84f5-a89c55076f65 2026-02-02 01:53:52,810 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:52,865 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:52,893 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/430babea-e844-4683-8da5-ec3534b72f04 2026-02-02 01:53:52,902 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/430babea-e844-4683-8da5-ec3534b72f04 2026-02-02 01:53:52,918 - DEBUG - Route Targets: [] 2026-02-02 01:53:52,918 - DEBUG - RT names not yet present for VN __public__ 2026-02-02 01:53:57,919 - DEBUG - Requesting: http://10.0.0.129:8082/domains 2026-02-02 01:53:57,926 - DEBUG - Requesting: http://10.0.0.129:8082/domain/7b9fb27b-8f56-456e-a3ed-374aed0f8bb9 2026-02-02 01:53:57,933 - DEBUG - Requesting: http://10.0.0.129:8082/project/294bb8df-e2a6-4dfe-84f5-a89c55076f65 2026-02-02 01:53:57,986 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:57,995 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:58,008 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/430babea-e844-4683-8da5-ec3534b72f04 2026-02-02 01:53:58,016 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/430babea-e844-4683-8da5-ec3534b72f04 2026-02-02 01:53:58,029 - DEBUG - Requesting: http://10.0.0.129:8082/route-target/e1f697fd-bc1e-4078-a7d9-d586ba60f014 2026-02-02 01:53:58,035 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-02 01:53:58,035 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:58,043 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/430babea-e844-4683-8da5-ec3534b72f04 2026-02-02 01:53:58,056 - INFO - Verified VN network id 13 for VN 24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:58,056 - INFO - Verifications in API Server for VN __public__ passed 2026-02-02 01:53:58,056 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:58,065 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/430babea-e844-4683-8da5-ec3534b72f04 2026-02-02 01:53:58,072 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/430babea-e844-4683-8da5-ec3534b72f04 2026-02-02 01:53:58,077 - DEBUG - Requesting: http://10.0.0.129:8082/route-target/e1f697fd-bc1e-4078-a7d9-d586ba60f014 2026-02-02 01:53:58,082 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-02 01:53:58,087 - DEBUG - Control-node 10.0.0.129 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2613315738628999417', 'uuid-lslong': '12992405653063184230'}, 'enable': 'true', 'created': '2026-02-02T01:53:52', 'last-modified': '2026-02-02T01:53:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.484766'} 2026-02-02 01:53:58,087 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-02 01:53:58,096 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-02 01:53:58,096 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-02-02 01:53:58,096 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-02-02 01:53:58,097 - DEBUG - Requesting: http://10.0.0.129:8082/domains 2026-02-02 01:53:58,100 - DEBUG - Requesting: http://10.0.0.129:8082/domain/7b9fb27b-8f56-456e-a3ed-374aed0f8bb9 2026-02-02 01:53:58,111 - DEBUG - Requesting: http://10.0.0.129:8082/project/294bb8df-e2a6-4dfe-84f5-a89c55076f65 2026-02-02 01:53:58,154 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:58,165 - DEBUG - =>VN __public__ has no policy to be verified 2026-02-02 01:53:58,165 - DEBUG - Verifying the vn in opserver 2026-02-02 01:53:58,165 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.129 2026-02-02 01:53:58,165 - DEBUG - Requesting: http://10.0.0.129:8081/analytics/uves/virtual-networks 2026-02-02 01:53:58,170 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.129:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-02-02 01:53:58,171 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-02-02 01:53:58,171 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-02 01:53:58,181 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-02 01:53:58,185 - DEBUG - Do not have enough data to verify VN in agent 2026-02-02 01:53:58,186 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-02 01:53:58,190 - DEBUG - VRF ids for VN __public__: {} 2026-02-02 01:53:58,279 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-02-02 01:53:58,383 - DEBUG - Requesting: http://10.0.0.129:8082/domains 2026-02-02 01:53:58,390 - DEBUG - Requesting: http://10.0.0.129:8082/domain/7b9fb27b-8f56-456e-a3ed-374aed0f8bb9 2026-02-02 01:53:58,397 - DEBUG - Requesting: http://10.0.0.129:8082/project/294bb8df-e2a6-4dfe-84f5-a89c55076f65 2026-02-02 01:53:58,439 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:58,450 - DEBUG - Requesting: http://10.0.0.129:8082/floating-ip-pool/f0b1ed8a-f4e3-4a29-b7e2-2d209ad6f0fe 2026-02-02 01:53:58,456 - DEBUG - Requesting: http://10.0.0.129:8082/domains 2026-02-02 01:53:58,460 - DEBUG - Requesting: http://10.0.0.129:8082/domain/7b9fb27b-8f56-456e-a3ed-374aed0f8bb9 2026-02-02 01:53:58,467 - DEBUG - Requesting: http://10.0.0.129:8082/project/294bb8df-e2a6-4dfe-84f5-a89c55076f65 2026-02-02 01:53:58,512 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/24445cd1-7eeb-48f9-b44e-4c1ad09baf66 2026-02-02 01:53:58,521 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-02-02 01:53:58,521 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-02-02 01:53:58,528 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-02 01:53:58,533 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-02-02 01:53:58,533 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-02-02 01:53:58,538 - DEBUG - Ingress ctest-nginx-ingress-30933373 not present 2026-02-02 01:53:58,538 - INFO - Creating Ingress ctest-nginx-ingress-30933373 2026-02-02 01:53:58,549 - INFO - Created Ingress ctest-nginx-ingress-30933373 2026-02-02 01:53:58,553 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-30933373 2026-02-02 01:54:01,558 - DEBUG - For Ingress ctest-nginx-ingress-30933373, Cluster IP: 10.47.255.249, External IPs ['165.240.239.252'] 2026-02-02 01:54:01,559 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-30933373 2026-02-02 01:54:01,559 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_LoadbalancerDatabaseList?x=3f3d359a-799f-4dd8-9c5c-b3fb841b74f2 2026-02-02 01:54:01,564 - INFO - Ingress ctest-nginx-ingress-30933373 with uuid 3f3d359a-799f-4dd8-9c5c-b3fb841b74f2 found in kube manager 2026-02-02 01:54:01,564 - INFO - Ingress ctest-nginx-ingress-30933373 verification passed 2026-02-02 01:54:01,567 - DEBUG - Pod ctest-busybox-pod-97501640 not present 2026-02-02 01:54:01,567 - INFO - Creating Pod ctest-busybox-pod-97501640 2026-02-02 01:54:01,613 - DEBUG - Pod : ctest-busybox-pod-97501640 UUID is be5f4105-48a0-48c1-b948-8aeea80f5b71 2026-02-02 01:54:01,622 - DEBUG - Pod ctest-nginx-pod-29093811 not in running state.Currently in Pending 2026-02-02 01:54:06,638 - DEBUG - Pod ctest-nginx-pod-29093811 not in running state.Currently in Pending 2026-02-02 01:54:11,648 - INFO - Pod ctest-nginx-pod-29093811 is in running state.Got IP 10.47.255.251 2026-02-02 01:54:11,667 - DEBUG - Pod ctest-nginx-pod-29093811 has vmi 072f02aa-ffda-11f0-8f85-fa163e48ddc5 2026-02-02 01:54:11,667 - INFO - Verified pod ctest-nginx-pod-29093811 in contrail-api 2026-02-02 01:54:11,668 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-02 01:54:11,681 - DEBUG - VMI 072f02aa-ffda-11f0-8f85-fa163e48ddc5 is active in agent 10.0.0.242 2026-02-02 01:54:11,681 - INFO - Verified Pod ctest-nginx-pod-29093811 in agent 10.0.0.242 2026-02-02 01:54:11,681 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_VirtualMachineDatabaseList?x=cf7bd11f-0237-4ff6-ab99-710d11cef5d8 2026-02-02 01:54:11,697 - INFO - Pod ctest-nginx-pod-29093811 with uuid cf7bd11f-0237-4ff6-ab99-710d11cef5d8 found in kube manager 2026-02-02 01:54:11,697 - INFO - Pod ctest-nginx-pod-29093811 verification passed 2026-02-02 01:54:11,778 - DEBUG - [Pod ctest-nginx-pod-29093811] Cmd: echo ctest-nginx-pod-29093811 > /usr/share/nginx/html/index.html, Output: 2026-02-02 01:54:11,793 - INFO - Pod ctest-nginx-pod-72027969 is in running state.Got IP 10.47.255.250 2026-02-02 01:54:11,856 - DEBUG - Pod ctest-nginx-pod-72027969 has vmi 07a62b6e-ffda-11f0-8f85-fa163e48ddc5 2026-02-02 01:54:11,856 - INFO - Verified pod ctest-nginx-pod-72027969 in contrail-api 2026-02-02 01:54:11,856 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-02 01:54:11,868 - DEBUG - VMI 07a62b6e-ffda-11f0-8f85-fa163e48ddc5 is active in agent 10.0.0.242 2026-02-02 01:54:11,868 - INFO - Verified Pod ctest-nginx-pod-72027969 in agent 10.0.0.242 2026-02-02 01:54:11,868 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_VirtualMachineDatabaseList?x=09cfce5a-38ec-48be-a689-20d1b04f649c 2026-02-02 01:54:11,875 - INFO - Pod ctest-nginx-pod-72027969 with uuid 09cfce5a-38ec-48be-a689-20d1b04f649c found in kube manager 2026-02-02 01:54:11,875 - INFO - Pod ctest-nginx-pod-72027969 verification passed 2026-02-02 01:54:11,945 - DEBUG - [Pod ctest-nginx-pod-72027969] Cmd: echo ctest-nginx-pod-72027969 > /usr/share/nginx/html/index.html, Output: 2026-02-02 01:54:11,955 - DEBUG - Pod ctest-busybox-pod-97501640 not in running state.Currently in Pending 2026-02-02 01:54:16,968 - INFO - Pod ctest-busybox-pod-97501640 is in running state.Got IP 10.47.255.248 2026-02-02 01:54:16,981 - DEBUG - Pod ctest-busybox-pod-97501640 has vmi 0c08ffc4-ffda-11f0-8f85-fa163e48ddc5 2026-02-02 01:54:16,981 - INFO - Verified pod ctest-busybox-pod-97501640 in contrail-api 2026-02-02 01:54:16,981 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-02 01:54:16,990 - DEBUG - VMI 0c08ffc4-ffda-11f0-8f85-fa163e48ddc5 is active in agent 10.0.0.242 2026-02-02 01:54:16,991 - INFO - Verified Pod ctest-busybox-pod-97501640 in agent 10.0.0.242 2026-02-02 01:54:16,991 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_VirtualMachineDatabaseList?x=be5f4105-48a0-48c1-b948-8aeea80f5b71 2026-02-02 01:54:16,994 - INFO - Pod ctest-busybox-pod-97501640 with uuid be5f4105-48a0-48c1-b948-8aeea80f5b71 found in kube manager 2026-02-02 01:54:16,994 - INFO - Pod ctest-busybox-pod-97501640 verification passed 2026-02-02 01:54:17,050 - DEBUG - [Pod ctest-busybox-pod-97501640] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-29093811 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-02 01:54:17,050 - DEBUG - [Pod Pod ctest-busybox-pod-97501640] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-02-02 01:54:17,050 - DEBUG - [Pod Pod ctest-busybox-pod-97501640] Cmd output: ctest-nginx-pod-29093811 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-02 01:54:17,101 - DEBUG - [Pod ctest-busybox-pod-97501640] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-72027969 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-02 01:54:17,101 - DEBUG - [Pod Pod ctest-busybox-pod-97501640] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-02-02 01:54:17,101 - DEBUG - [Pod Pod ctest-busybox-pod-97501640] Cmd output: ctest-nginx-pod-72027969 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-02 01:54:17,101 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-29093811': 1, 'ctest-nginx-pod-72027969': 1} 2026-02-02 01:54:17,101 - WARNING - Nothing to delete parallely 2026-02-02 01:54:17,102 - INFO - Deleting pod default:ctest-busybox-pod-97501640 2026-02-02 01:54:17,116 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=be5f4105-48a0-48c1-b948-8aeea80f5b71 2026-02-02 01:54:17,122 - WARNING - Pod uuid be5f4105-48a0-48c1-b948-8aeea80f5b71 is still seen in agent 10.0.0.242 VM list 2026-02-02 01:54:22,122 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=be5f4105-48a0-48c1-b948-8aeea80f5b71 2026-02-02 01:54:22,128 - WARNING - Pod uuid be5f4105-48a0-48c1-b948-8aeea80f5b71 is still seen in agent 10.0.0.242 VM list 2026-02-02 01:54:27,129 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=be5f4105-48a0-48c1-b948-8aeea80f5b71 2026-02-02 01:54:27,136 - WARNING - Pod uuid be5f4105-48a0-48c1-b948-8aeea80f5b71 is still seen in agent 10.0.0.242 VM list 2026-02-02 01:54:32,137 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=be5f4105-48a0-48c1-b948-8aeea80f5b71 2026-02-02 01:54:32,145 - WARNING - Pod uuid be5f4105-48a0-48c1-b948-8aeea80f5b71 is still seen in agent 10.0.0.242 VM list 2026-02-02 01:54:37,146 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=be5f4105-48a0-48c1-b948-8aeea80f5b71 2026-02-02 01:54:37,152 - WARNING - Pod uuid be5f4105-48a0-48c1-b948-8aeea80f5b71 is still seen in agent 10.0.0.242 VM list 2026-02-02 01:54:42,153 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=be5f4105-48a0-48c1-b948-8aeea80f5b71 2026-02-02 01:54:42,159 - WARNING - Pod uuid be5f4105-48a0-48c1-b948-8aeea80f5b71 is still seen in agent 10.0.0.242 VM list 2026-02-02 01:54:47,160 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=be5f4105-48a0-48c1-b948-8aeea80f5b71 2026-02-02 01:54:47,167 - WARNING - Pod uuid be5f4105-48a0-48c1-b948-8aeea80f5b71 is still seen in agent 10.0.0.242 VM list 2026-02-02 01:54:52,167 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=be5f4105-48a0-48c1-b948-8aeea80f5b71 2026-02-02 01:54:52,174 - DEBUG - Pod be5f4105-48a0-48c1-b948-8aeea80f5b71 is not in agent 10.0.0.242 VM list 2026-02-02 01:54:52,174 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:0c08ffc4-ffda-11f0-8f85-fa163e48ddc5 2026-02-02 01:54:52,180 - DEBUG - VMI 0c08ffc4-ffda-11f0-8f85-fa163e48ddc5 is removed from agent 10.0.0.242 2026-02-02 01:54:52,180 - INFO - Verified that pod ctest-busybox-pod-97501640 is removed in agent 2026-02-02 01:54:52,180 - INFO - Deleting Ingress : ctest-nginx-ingress-30933373 2026-02-02 01:54:52,190 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_LoadbalancerDatabaseList?x=3f3d359a-799f-4dd8-9c5c-b3fb841b74f2 2026-02-02 01:54:52,194 - ERROR - Ingress ctest-nginx-ingress-30933373 with uuid 3f3d359a-799f-4dd8-9c5c-b3fb841b74f2 still found in kube manager 2026-02-02 01:54:53,195 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_LoadbalancerDatabaseList?x=3f3d359a-799f-4dd8-9c5c-b3fb841b74f2 2026-02-02 01:54:53,199 - INFO - Ingress ctest-nginx-ingress-30933373 with uuid 3f3d359a-799f-4dd8-9c5c-b3fb841b74f2 deleted successfully from kube manager 2026-02-02 01:54:53,199 - INFO - Verifications on Ingress %s cleanup passed 2026-02-02 01:54:53,199 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-02-02 01:54:53,347 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-02 01:54:53,357 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-02-02 01:54:53,358 - INFO - Deleting VN __public__ 2026-02-02 01:54:53,614 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/430babea-e844-4683-8da5-ec3534b72f04 2026-02-02 01:54:53,626 - DEBUG - Response Code: 404 2026-02-02 01:54:53,626 - DEBUG - Requesting: http://10.0.0.129:8082/domains 2026-02-02 01:54:53,631 - DEBUG - Requesting: http://10.0.0.129:8082/domain/7b9fb27b-8f56-456e-a3ed-374aed0f8bb9 2026-02-02 01:54:53,640 - DEBUG - Requesting: http://10.0.0.129:8082/project/294bb8df-e2a6-4dfe-84f5-a89c55076f65 2026-02-02 01:54:53,723 - INFO - Validated that VN __public__ is not found in API Server 2026-02-02 01:54:53,723 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-02-02 01:54:53,731 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-02 01:54:53,743 - DEBUG - VN __public__ is not present in Agent 10.0.0.242 2026-02-02 01:54:53,743 - INFO - Validated that VN __public__ is not in any agent 2026-02-02 01:54:53,743 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-02 01:54:53,763 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-02 01:54:53,782 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-02 01:54:53,800 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-02-02 01:54:53,801 - INFO - Deleting pod default:ctest-nginx-pod-72027969 2026-02-02 01:54:53,822 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=09cfce5a-38ec-48be-a689-20d1b04f649c 2026-02-02 01:54:53,848 - WARNING - Pod uuid 09cfce5a-38ec-48be-a689-20d1b04f649c is still seen in agent 10.0.0.242 VM list 2026-02-02 01:54:58,848 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=09cfce5a-38ec-48be-a689-20d1b04f649c 2026-02-02 01:54:58,855 - DEBUG - Pod 09cfce5a-38ec-48be-a689-20d1b04f649c is not in agent 10.0.0.242 VM list 2026-02-02 01:54:58,855 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:07a62b6e-ffda-11f0-8f85-fa163e48ddc5 2026-02-02 01:54:58,861 - DEBUG - VMI 07a62b6e-ffda-11f0-8f85-fa163e48ddc5 is removed from agent 10.0.0.242 2026-02-02 01:54:58,861 - INFO - Verified that pod ctest-nginx-pod-72027969 is removed in agent 2026-02-02 01:54:58,861 - INFO - Deleting pod default:ctest-nginx-pod-29093811 2026-02-02 01:54:58,872 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=cf7bd11f-0237-4ff6-ab99-710d11cef5d8 2026-02-02 01:54:58,880 - WARNING - Pod uuid cf7bd11f-0237-4ff6-ab99-710d11cef5d8 is still seen in agent 10.0.0.242 VM list 2026-02-02 01:55:03,881 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=cf7bd11f-0237-4ff6-ab99-710d11cef5d8 2026-02-02 01:55:03,889 - DEBUG - Pod cf7bd11f-0237-4ff6-ab99-710d11cef5d8 is not in agent 10.0.0.242 VM list 2026-02-02 01:55:03,889 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:072f02aa-ffda-11f0-8f85-fa163e48ddc5 2026-02-02 01:55:03,895 - DEBUG - VMI 072f02aa-ffda-11f0-8f85-fa163e48ddc5 is removed from agent 10.0.0.242 2026-02-02 01:55:03,895 - INFO - Verified that pod ctest-nginx-pod-29093811 is removed in agent 2026-02-02 01:55:03,896 - INFO - Deleting service : ctest-nginx-svc-07227689 2026-02-02 01:55:04,182 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-02 01:55:04,191 - DEBUG - No XMPP flaps were noticed during the test