2026-03-24 20:21:12,980 - DEBUG - [10.0.0.19]: Running cmd : hostname 2026-03-24 20:21:13,227 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1955-1 2026-03-24 20:21:13,227 - DEBUG - [10.0.0.19]: Running cmd : hostname -f 2026-03-24 20:21:13,262 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1955-1 2026-03-24 20:21:13,262 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-03-24 20:21:13,401 - DEBUG - Output : NAMES contrail_test_7tvpSHtAe contrail-kubernetes-master-kubemanager-1 analytics_database-cassandra-1 analytics_database-query-engine-1 analytics_database-nodemgr-1 analytics_snmp-nodemgr-1 analytics_snmp-topology-1 analytics_snmp-snmp-collector-1 analytics_alarm-nodemgr-1 analytics_alarm-alarm-gen-1 analytics_alarm-kafka-1 analytics-nodemgr-1 analytics-api-1 analytics-collector-1 redis-redis-1 webui-web-1 webui-job-1 control-named-1 control-nodemgr-1 control-control-1 control-dns-1 config_database-rabbitmq-1 config_database-nodemgr-1 config_database-zookeeper-1 config_database-cassandra-1 config_api-devicemgr-1 config_api-nodemgr-1 config_api-api-1 config_api-svcmonitor-1 config_api-schema-1 config_api-dnsmasq-1 2026-03-24 20:21:13,401 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-03-24 20:21:13,437 - DEBUG - Output : 127.0.0.1/8 10.0.0.19/24 252.19.0.1/8 10.2.48.1/24 172.17.0.1/16 2026-03-24 20:21:13,437 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.0.0.19 | head -n 1 | awk '{print $2}' 2026-03-24 20:21:13,471 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1955-1 2026-03-24 20:21:13,556 - DEBUG - [10.0.0.41]: Running cmd : hostname 2026-03-24 20:21:13,777 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1955-1 2026-03-24 20:21:13,777 - DEBUG - [10.0.0.41]: Running cmd : hostname -f 2026-03-24 20:21:13,811 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1955-1 2026-03-24 20:21:13,811 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-03-24 20:21:13,856 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2026-03-24 20:21:13,856 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-03-24 20:21:13,890 - DEBUG - Output : 10.0.0.41/24 2026-03-24 20:21:13,890 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.0.0.41 | head -n 1 | awk '{print $2}' 2026-03-24 20:21:13,959 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1955-1 2026-03-24 20:21:14,029 - INFO - Using existing project ['default-domain', 'k8s-default'](c2276194-4173-4e58-9140-b38f464ad03b) 2026-03-24 20:21:14,098 - INFO - Using existing project ['default-domain', 'k8s-default'](c2276194-4173-4e58-9140-b38f464ad03b) 2026-03-24 20:21:14,200 - INFO - ================================================================================ 2026-03-24 20:21:14,200 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-03-24 20:21:14,200 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-03-24 20:21:14,442 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-24 20:21:14,448 - DEBUG - Nothing to compare xmpp stats {'10.0.0.41': {'10.0.0.19': '0'}} with 2026-03-24 20:21:14,448 - INFO - Initial checks done. Running the testcase now 2026-03-24 20:21:14,448 - INFO - 2026-03-24 20:21:14,452 - INFO - Namespace default already exists 2026-03-24 20:21:14,453 - DEBUG - [10.0.0.19]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-03-24 20:21:14,558 - DEBUG - Output : cluster_project={} 2026-03-24 20:21:14,647 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-03-24 20:21:14,647 - DEBUG - Project uuid in Contrail is c2276194-4173-4e58-9140-b38f464ad03b 2026-03-24 20:21:14,647 - DEBUG - [10.0.0.19]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-03-24 20:21:14,786 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-03-24 20:21:14,787 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_NamespaceDatabaseList?namespace_uuid=3b4c434c-c1cf-4ae9-9539-70bb7e46c39b 2026-03-24 20:21:14,791 - INFO - Namespace default with uuid 3b4c434c-c1cf-4ae9-9539-70bb7e46c39b found in kube manager 2026-03-24 20:21:14,791 - INFO - Namespace default verification passed 2026-03-24 20:21:14,794 - DEBUG - Service ctest-nginx-svc-06073647 not present 2026-03-24 20:21:14,794 - INFO - Creating service ctest-nginx-svc-06073647 2026-03-24 20:21:14,810 - INFO - Created Service ctest-nginx-svc-06073647 2026-03-24 20:21:14,813 - DEBUG - Pod ctest-nginx-pod-34903799 not present 2026-03-24 20:21:14,814 - INFO - Creating Pod ctest-nginx-pod-34903799 2026-03-24 20:21:14,887 - DEBUG - Pod : ctest-nginx-pod-34903799 UUID is 3a205710-a362-43e8-a321-1ccd672f26cd 2026-03-24 20:21:14,890 - DEBUG - Pod ctest-nginx-pod-02723440 not present 2026-03-24 20:21:14,891 - INFO - Creating Pod ctest-nginx-pod-02723440 2026-03-24 20:21:14,927 - DEBUG - Pod : ctest-nginx-pod-02723440 UUID is c4dcc182-6f72-4efe-ad23-1efbb9042880 2026-03-24 20:21:15,097 - INFO - Created VN __public__, UUID :84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:15,230 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2026-03-24 20:21:15,235 - DEBUG - Requesting: http://10.0.0.19:8082/domain/ea1b87ea-ebb2-4509-be25-87fb5b5e92ad 2026-03-24 20:21:15,277 - DEBUG - Requesting: http://10.0.0.19:8082/project/c2276194-4173-4e58-9140-b38f464ad03b 2026-03-24 20:21:15,334 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:15,348 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:15,359 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e6e87f53-2538-4ae5-805f-4b64c29966b1 2026-03-24 20:21:15,367 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e6e87f53-2538-4ae5-805f-4b64c29966b1 2026-03-24 20:21:15,372 - DEBUG - Route Targets: [] 2026-03-24 20:21:15,372 - DEBUG - RT names not yet present for VN __public__ 2026-03-24 20:21:20,374 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2026-03-24 20:21:20,380 - DEBUG - Requesting: http://10.0.0.19:8082/domain/ea1b87ea-ebb2-4509-be25-87fb5b5e92ad 2026-03-24 20:21:20,386 - DEBUG - Requesting: http://10.0.0.19:8082/project/c2276194-4173-4e58-9140-b38f464ad03b 2026-03-24 20:21:20,427 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:20,435 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:20,442 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e6e87f53-2538-4ae5-805f-4b64c29966b1 2026-03-24 20:21:20,452 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e6e87f53-2538-4ae5-805f-4b64c29966b1 2026-03-24 20:21:20,457 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/2d0c6de4-8a99-41e8-8093-f520f4d47b6c 2026-03-24 20:21:20,462 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-24 20:21:20,462 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:20,468 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e6e87f53-2538-4ae5-805f-4b64c29966b1 2026-03-24 20:21:20,477 - INFO - Verified VN network id 13 for VN 84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:20,477 - INFO - Verifications in API Server for VN __public__ passed 2026-03-24 20:21:20,477 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:20,484 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e6e87f53-2538-4ae5-805f-4b64c29966b1 2026-03-24 20:21:20,488 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e6e87f53-2538-4ae5-805f-4b64c29966b1 2026-03-24 20:21:20,494 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/2d0c6de4-8a99-41e8-8093-f520f4d47b6c 2026-03-24 20:21:20,500 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-24 20:21:20,505 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9570982396461925675', 'uuid-lslong': '13529932794690621220'}, 'enable': 'true', 'created': '2026-03-24T20:21:15', 'last-modified': '2026-03-24T20:21:15', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.371324'} 2026-03-24 20:21:20,505 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 20:21:20,515 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-24 20:21:20,515 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-03-24 20:21:20,515 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-03-24 20:21:20,515 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2026-03-24 20:21:20,520 - DEBUG - Requesting: http://10.0.0.19:8082/domain/ea1b87ea-ebb2-4509-be25-87fb5b5e92ad 2026-03-24 20:21:20,526 - DEBUG - Requesting: http://10.0.0.19:8082/project/c2276194-4173-4e58-9140-b38f464ad03b 2026-03-24 20:21:20,566 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:20,575 - DEBUG - =>VN __public__ has no policy to be verified 2026-03-24 20:21:20,575 - DEBUG - Verifying the vn in opserver 2026-03-24 20:21:20,575 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.19 2026-03-24 20:21:20,575 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2026-03-24 20:21:20,580 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-03-24 20:21:20,580 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-03-24 20:21:20,580 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 20:21:20,588 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 20:21:20,593 - DEBUG - Do not have enough data to verify VN in agent 2026-03-24 20:21:20,593 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 20:21:20,598 - DEBUG - VRF ids for VN __public__: {} 2026-03-24 20:21:20,694 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-03-24 20:21:20,806 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2026-03-24 20:21:20,813 - DEBUG - Requesting: http://10.0.0.19:8082/domain/ea1b87ea-ebb2-4509-be25-87fb5b5e92ad 2026-03-24 20:21:20,821 - DEBUG - Requesting: http://10.0.0.19:8082/project/c2276194-4173-4e58-9140-b38f464ad03b 2026-03-24 20:21:20,862 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:20,877 - DEBUG - Requesting: http://10.0.0.19:8082/floating-ip-pool/683dba40-2d10-4cae-bb8f-93cc4f46d22b 2026-03-24 20:21:20,883 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2026-03-24 20:21:20,887 - DEBUG - Requesting: http://10.0.0.19:8082/domain/ea1b87ea-ebb2-4509-be25-87fb5b5e92ad 2026-03-24 20:21:20,893 - DEBUG - Requesting: http://10.0.0.19:8082/project/c2276194-4173-4e58-9140-b38f464ad03b 2026-03-24 20:21:20,945 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/84d2f5c7-c4a1-452b-bbc3-fa312e714f24 2026-03-24 20:21:20,952 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-03-24 20:21:20,952 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-03-24 20:21:20,959 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-24 20:21:20,964 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-03-24 20:21:20,964 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-03-24 20:21:20,967 - DEBUG - Ingress ctest-nginx-ingress-25298283 not present 2026-03-24 20:21:20,968 - INFO - Creating Ingress ctest-nginx-ingress-25298283 2026-03-24 20:21:20,975 - INFO - Created Ingress ctest-nginx-ingress-25298283 2026-03-24 20:21:20,979 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-25298283 2026-03-24 20:21:23,984 - DEBUG - For Ingress ctest-nginx-ingress-25298283, Cluster IP: 10.47.255.249, External IPs ['218.207.189.60'] 2026-03-24 20:21:23,984 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-25298283 2026-03-24 20:21:23,985 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_LoadbalancerDatabaseList?x=8f49573a-e0d0-4c3d-81db-83a075189c49 2026-03-24 20:21:23,993 - INFO - Ingress ctest-nginx-ingress-25298283 with uuid 8f49573a-e0d0-4c3d-81db-83a075189c49 found in kube manager 2026-03-24 20:21:23,993 - INFO - Ingress ctest-nginx-ingress-25298283 verification passed 2026-03-24 20:21:23,997 - DEBUG - Pod ctest-busybox-pod-39064735 not present 2026-03-24 20:21:23,997 - INFO - Creating Pod ctest-busybox-pod-39064735 2026-03-24 20:21:24,059 - DEBUG - Pod : ctest-busybox-pod-39064735 UUID is 5a3eaa0c-9884-4fab-a6af-6f295214ca05 2026-03-24 20:21:24,084 - DEBUG - Pod ctest-nginx-pod-34903799 not in running state.Currently in Pending 2026-03-24 20:21:29,133 - INFO - Pod ctest-nginx-pod-34903799 is in running state.Got IP 10.47.255.251 2026-03-24 20:21:29,146 - DEBUG - Pod ctest-nginx-pod-34903799 has vmi 02d298fe-27bf-11f1-9fa4-fa163e64f847 2026-03-24 20:21:29,146 - INFO - Verified pod ctest-nginx-pod-34903799 in contrail-api 2026-03-24 20:21:29,146 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-24 20:21:29,161 - DEBUG - VMI 02d298fe-27bf-11f1-9fa4-fa163e64f847 is active in agent 10.0.0.41 2026-03-24 20:21:29,161 - INFO - Verified Pod ctest-nginx-pod-34903799 in agent 10.0.0.41 2026-03-24 20:21:29,161 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_VirtualMachineDatabaseList?x=3a205710-a362-43e8-a321-1ccd672f26cd 2026-03-24 20:21:29,173 - INFO - Pod ctest-nginx-pod-34903799 with uuid 3a205710-a362-43e8-a321-1ccd672f26cd found in kube manager 2026-03-24 20:21:29,173 - INFO - Pod ctest-nginx-pod-34903799 verification passed 2026-03-24 20:21:29,250 - DEBUG - [Pod ctest-nginx-pod-34903799] Cmd: echo ctest-nginx-pod-34903799 > /usr/share/nginx/html/index.html, Output: 2026-03-24 20:21:29,265 - INFO - Pod ctest-nginx-pod-02723440 is in running state.Got IP 10.47.255.250 2026-03-24 20:21:29,282 - DEBUG - Pod ctest-nginx-pod-02723440 has vmi 033aa5de-27bf-11f1-9fa4-fa163e64f847 2026-03-24 20:21:29,282 - INFO - Verified pod ctest-nginx-pod-02723440 in contrail-api 2026-03-24 20:21:29,283 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-24 20:21:29,292 - DEBUG - VMI 033aa5de-27bf-11f1-9fa4-fa163e64f847 is active in agent 10.0.0.41 2026-03-24 20:21:29,293 - INFO - Verified Pod ctest-nginx-pod-02723440 in agent 10.0.0.41 2026-03-24 20:21:29,293 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_VirtualMachineDatabaseList?x=c4dcc182-6f72-4efe-ad23-1efbb9042880 2026-03-24 20:21:29,297 - INFO - Pod ctest-nginx-pod-02723440 with uuid c4dcc182-6f72-4efe-ad23-1efbb9042880 found in kube manager 2026-03-24 20:21:29,297 - INFO - Pod ctest-nginx-pod-02723440 verification passed 2026-03-24 20:21:29,358 - DEBUG - [Pod ctest-nginx-pod-02723440] Cmd: echo ctest-nginx-pod-02723440 > /usr/share/nginx/html/index.html, Output: 2026-03-24 20:21:29,373 - DEBUG - Pod ctest-busybox-pod-39064735 not in running state.Currently in Pending 2026-03-24 20:21:34,383 - DEBUG - Pod ctest-busybox-pod-39064735 not in running state.Currently in Pending 2026-03-24 20:21:39,390 - INFO - Pod ctest-busybox-pod-39064735 is in running state.Got IP 10.47.255.248 2026-03-24 20:21:39,450 - DEBUG - Pod ctest-busybox-pod-39064735 has vmi 07f00218-27bf-11f1-9fa4-fa163e64f847 2026-03-24 20:21:39,450 - INFO - Verified pod ctest-busybox-pod-39064735 in contrail-api 2026-03-24 20:21:39,450 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-24 20:21:39,459 - DEBUG - VMI 07f00218-27bf-11f1-9fa4-fa163e64f847 is active in agent 10.0.0.41 2026-03-24 20:21:39,459 - INFO - Verified Pod ctest-busybox-pod-39064735 in agent 10.0.0.41 2026-03-24 20:21:39,459 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_VirtualMachineDatabaseList?x=5a3eaa0c-9884-4fab-a6af-6f295214ca05 2026-03-24 20:21:39,463 - INFO - Pod ctest-busybox-pod-39064735 with uuid 5a3eaa0c-9884-4fab-a6af-6f295214ca05 found in kube manager 2026-03-24 20:21:39,463 - INFO - Pod ctest-busybox-pod-39064735 verification passed 2026-03-24 20:21:39,518 - DEBUG - [Pod ctest-busybox-pod-39064735] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) ctest-nginx-pod-02723440 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-24 20:21:39,518 - DEBUG - [Pod Pod ctest-busybox-pod-39064735] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-03-24 20:21:39,518 - DEBUG - [Pod Pod ctest-busybox-pod-39064735] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) ctest-nginx-pod-02723440 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-24 20:21:39,599 - DEBUG - [Pod ctest-busybox-pod-39064735] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-02723440 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-24 20:21:39,599 - DEBUG - [Pod Pod ctest-busybox-pod-39064735] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-03-24 20:21:39,599 - DEBUG - [Pod Pod ctest-busybox-pod-39064735] Cmd output: ctest-nginx-pod-02723440 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-24 20:21:39,650 - DEBUG - [Pod ctest-busybox-pod-39064735] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-34903799 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-24 20:21:39,650 - DEBUG - [Pod Pod ctest-busybox-pod-39064735] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-03-24 20:21:39,650 - DEBUG - [Pod Pod ctest-busybox-pod-39064735] Cmd output: ctest-nginx-pod-34903799 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-24 20:21:39,650 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-34903799': 1, 'ctest-nginx-pod-02723440': 2} 2026-03-24 20:21:39,650 - WARNING - Nothing to delete parallely 2026-03-24 20:21:39,650 - INFO - Deleting pod default:ctest-busybox-pod-39064735 2026-03-24 20:21:39,667 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=5a3eaa0c-9884-4fab-a6af-6f295214ca05 2026-03-24 20:21:39,675 - WARNING - Pod uuid 5a3eaa0c-9884-4fab-a6af-6f295214ca05 is still seen in agent 10.0.0.41 VM list 2026-03-24 20:21:44,676 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=5a3eaa0c-9884-4fab-a6af-6f295214ca05 2026-03-24 20:21:44,688 - WARNING - Pod uuid 5a3eaa0c-9884-4fab-a6af-6f295214ca05 is still seen in agent 10.0.0.41 VM list 2026-03-24 20:21:49,689 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=5a3eaa0c-9884-4fab-a6af-6f295214ca05 2026-03-24 20:21:49,696 - WARNING - Pod uuid 5a3eaa0c-9884-4fab-a6af-6f295214ca05 is still seen in agent 10.0.0.41 VM list 2026-03-24 20:21:54,697 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=5a3eaa0c-9884-4fab-a6af-6f295214ca05 2026-03-24 20:21:54,705 - WARNING - Pod uuid 5a3eaa0c-9884-4fab-a6af-6f295214ca05 is still seen in agent 10.0.0.41 VM list 2026-03-24 20:21:59,705 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=5a3eaa0c-9884-4fab-a6af-6f295214ca05 2026-03-24 20:21:59,722 - WARNING - Pod uuid 5a3eaa0c-9884-4fab-a6af-6f295214ca05 is still seen in agent 10.0.0.41 VM list 2026-03-24 20:22:04,723 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=5a3eaa0c-9884-4fab-a6af-6f295214ca05 2026-03-24 20:22:04,729 - WARNING - Pod uuid 5a3eaa0c-9884-4fab-a6af-6f295214ca05 is still seen in agent 10.0.0.41 VM list 2026-03-24 20:22:09,730 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=5a3eaa0c-9884-4fab-a6af-6f295214ca05 2026-03-24 20:22:09,739 - WARNING - Pod uuid 5a3eaa0c-9884-4fab-a6af-6f295214ca05 is still seen in agent 10.0.0.41 VM list 2026-03-24 20:22:14,739 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=5a3eaa0c-9884-4fab-a6af-6f295214ca05 2026-03-24 20:22:14,746 - DEBUG - Pod 5a3eaa0c-9884-4fab-a6af-6f295214ca05 is not in agent 10.0.0.41 VM list 2026-03-24 20:22:14,746 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:07f00218-27bf-11f1-9fa4-fa163e64f847 2026-03-24 20:22:14,751 - DEBUG - VMI 07f00218-27bf-11f1-9fa4-fa163e64f847 is removed from agent 10.0.0.41 2026-03-24 20:22:14,751 - INFO - Verified that pod ctest-busybox-pod-39064735 is removed in agent 2026-03-24 20:22:14,752 - INFO - Deleting Ingress : ctest-nginx-ingress-25298283 2026-03-24 20:22:14,767 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_LoadbalancerDatabaseList?x=8f49573a-e0d0-4c3d-81db-83a075189c49 2026-03-24 20:22:14,772 - ERROR - Ingress ctest-nginx-ingress-25298283 with uuid 8f49573a-e0d0-4c3d-81db-83a075189c49 still found in kube manager 2026-03-24 20:22:15,772 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_LoadbalancerDatabaseList?x=8f49573a-e0d0-4c3d-81db-83a075189c49 2026-03-24 20:22:15,776 - INFO - Ingress ctest-nginx-ingress-25298283 with uuid 8f49573a-e0d0-4c3d-81db-83a075189c49 deleted successfully from kube manager 2026-03-24 20:22:15,776 - INFO - Verifications on Ingress %s cleanup passed 2026-03-24 20:22:15,776 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-03-24 20:22:15,959 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-24 20:22:15,971 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-03-24 20:22:15,972 - INFO - Deleting VN __public__ 2026-03-24 20:22:16,152 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/e6e87f53-2538-4ae5-805f-4b64c29966b1 2026-03-24 20:22:16,156 - DEBUG - Response Code: 404 2026-03-24 20:22:16,156 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2026-03-24 20:22:16,171 - DEBUG - Requesting: http://10.0.0.19:8082/domain/ea1b87ea-ebb2-4509-be25-87fb5b5e92ad 2026-03-24 20:22:16,184 - DEBUG - Requesting: http://10.0.0.19:8082/project/c2276194-4173-4e58-9140-b38f464ad03b 2026-03-24 20:22:16,246 - INFO - Validated that VN __public__ is not found in API Server 2026-03-24 20:22:16,246 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-03-24 20:22:16,258 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 20:22:16,265 - DEBUG - VN __public__ is not present in Agent 10.0.0.41 2026-03-24 20:22:16,265 - INFO - Validated that VN __public__ is not in any agent 2026-03-24 20:22:16,265 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 20:22:16,275 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 20:22:16,305 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-24 20:22:16,310 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-03-24 20:22:16,310 - INFO - Deleting pod default:ctest-nginx-pod-02723440 2026-03-24 20:22:16,332 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=c4dcc182-6f72-4efe-ad23-1efbb9042880 2026-03-24 20:22:16,349 - WARNING - Pod uuid c4dcc182-6f72-4efe-ad23-1efbb9042880 is still seen in agent 10.0.0.41 VM list 2026-03-24 20:22:21,351 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=c4dcc182-6f72-4efe-ad23-1efbb9042880 2026-03-24 20:22:21,357 - DEBUG - Pod c4dcc182-6f72-4efe-ad23-1efbb9042880 is not in agent 10.0.0.41 VM list 2026-03-24 20:22:21,357 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:033aa5de-27bf-11f1-9fa4-fa163e64f847 2026-03-24 20:22:21,363 - DEBUG - VMI 033aa5de-27bf-11f1-9fa4-fa163e64f847 is removed from agent 10.0.0.41 2026-03-24 20:22:21,363 - INFO - Verified that pod ctest-nginx-pod-02723440 is removed in agent 2026-03-24 20:22:21,364 - INFO - Deleting pod default:ctest-nginx-pod-34903799 2026-03-24 20:22:21,380 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=3a205710-a362-43e8-a321-1ccd672f26cd 2026-03-24 20:22:21,398 - WARNING - Pod uuid 3a205710-a362-43e8-a321-1ccd672f26cd is still seen in agent 10.0.0.41 VM list 2026-03-24 20:22:26,398 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=3a205710-a362-43e8-a321-1ccd672f26cd 2026-03-24 20:22:26,405 - DEBUG - Pod 3a205710-a362-43e8-a321-1ccd672f26cd is not in agent 10.0.0.41 VM list 2026-03-24 20:22:26,405 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:02d298fe-27bf-11f1-9fa4-fa163e64f847 2026-03-24 20:22:26,410 - DEBUG - VMI 02d298fe-27bf-11f1-9fa4-fa163e64f847 is removed from agent 10.0.0.41 2026-03-24 20:22:26,410 - INFO - Verified that pod ctest-nginx-pod-34903799 is removed in agent 2026-03-24 20:22:26,410 - INFO - Deleting service : ctest-nginx-svc-06073647 2026-03-24 20:22:26,613 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-24 20:22:26,619 - DEBUG - No XMPP flaps were noticed during the test