2026-03-17 15:01:38,157 - DEBUG - [10.0.0.50]: Running cmd : hostname 2026-03-17 15:01:38,541 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1932-1 2026-03-17 15:01:38,542 - DEBUG - [10.0.0.50]: Running cmd : hostname -f 2026-03-17 15:01:38,587 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1932-1 2026-03-17 15:01:38,587 - DEBUG - [10.0.0.50]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-03-17 15:01:38,694 - DEBUG - Output : NAMES contrail_test_RZHKCswoi contrail-kubernetes-master-kubemanager-1 analytics_snmp-nodemgr-1 analytics_snmp-snmp-collector-1 analytics_snmp-topology-1 analytics_alarm-kafka-1 analytics_alarm-alarm-gen-1 analytics_alarm-nodemgr-1 analytics-collector-1 analytics-nodemgr-1 analytics-api-1 analytics_database-query-engine-1 analytics_database-nodemgr-1 analytics_database-cassandra-1 redis-redis-1 webui-job-1 webui-web-1 control-named-1 control-control-1 control-nodemgr-1 control-dns-1 config_database-rabbitmq-1 config_database-cassandra-1 config_database-zookeeper-1 config_database-nodemgr-1 config_api-devicemgr-1 config_api-svcmonitor-1 config_api-api-1 config_api-nodemgr-1 config_api-schema-1 config_api-dnsmasq-1 2026-03-17 15:01:38,695 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-03-17 15:01:38,786 - DEBUG - Output : 127.0.0.1/8 10.0.0.50/24 252.50.0.1/8 10.100.83.1/24 172.17.0.1/16 2026-03-17 15:01:38,787 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.0.0.50 | head -n 1 | awk '{print $2}' 2026-03-17 15:01:38,843 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1932-1 2026-03-17 15:01:38,962 - DEBUG - [10.0.0.33]: Running cmd : hostname 2026-03-17 15:01:39,241 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1932-1 2026-03-17 15:01:39,241 - DEBUG - [10.0.0.33]: Running cmd : hostname -f 2026-03-17 15:01:39,269 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1932-1 2026-03-17 15:01:39,269 - DEBUG - [10.0.0.33]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-03-17 15:01:39,364 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2026-03-17 15:01:39,365 - DEBUG - [10.0.0.33]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-03-17 15:01:39,402 - DEBUG - Output : 10.0.0.33/24 2026-03-17 15:01:39,403 - DEBUG - [10.0.0.33]: Running cmd : getent hosts 10.0.0.33 | head -n 1 | awk '{print $2}' 2026-03-17 15:01:39,438 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1932-1 2026-03-17 15:01:39,597 - INFO - Using existing project ['default-domain', 'k8s-default'](9f9654e6-9f79-4b99-80a9-d26eacded17b) 2026-03-17 15:01:39,742 - INFO - Using existing project ['default-domain', 'k8s-default'](9f9654e6-9f79-4b99-80a9-d26eacded17b) 2026-03-17 15:01:39,871 - INFO - ================================================================================ 2026-03-17 15:01:39,871 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-03-17 15:01:39,872 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-03-17 15:01:40,150 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-17 15:01:40,159 - DEBUG - Nothing to compare xmpp stats {'10.0.0.33': {'10.0.0.50': '0'}} with 2026-03-17 15:01:40,159 - INFO - Initial checks done. Running the testcase now 2026-03-17 15:01:40,159 - INFO - 2026-03-17 15:01:40,166 - INFO - Namespace default already exists 2026-03-17 15:01:40,167 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-03-17 15:01:40,328 - DEBUG - Output : cluster_project={} 2026-03-17 15:01:40,405 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-03-17 15:01:40,406 - DEBUG - Project uuid in Contrail is 9f9654e6-9f79-4b99-80a9-d26eacded17b 2026-03-17 15:01:40,406 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-03-17 15:01:40,584 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-03-17 15:01:40,584 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_NamespaceDatabaseList?namespace_uuid=5226d978-8cef-4107-a4d5-6073d9982dcc 2026-03-17 15:01:40,591 - INFO - Namespace default with uuid 5226d978-8cef-4107-a4d5-6073d9982dcc found in kube manager 2026-03-17 15:01:40,592 - INFO - Namespace default verification passed 2026-03-17 15:01:40,596 - DEBUG - Service ctest-nginx-svc-03129834 not present 2026-03-17 15:01:40,597 - INFO - Creating service ctest-nginx-svc-03129834 2026-03-17 15:01:40,620 - INFO - Created Service ctest-nginx-svc-03129834 2026-03-17 15:01:40,640 - DEBUG - Pod ctest-nginx-pod-13666412 not present 2026-03-17 15:01:40,643 - INFO - Creating Pod ctest-nginx-pod-13666412 2026-03-17 15:01:40,739 - DEBUG - Pod : ctest-nginx-pod-13666412 UUID is af114e27-b115-4c01-b6ae-10bdf412a1e5 2026-03-17 15:01:40,758 - DEBUG - Pod ctest-nginx-pod-92705561 not present 2026-03-17 15:01:40,758 - INFO - Creating Pod ctest-nginx-pod-92705561 2026-03-17 15:01:40,795 - DEBUG - Pod : ctest-nginx-pod-92705561 UUID is 8d634122-d636-4a88-b1f3-5e04787b0a30 2026-03-17 15:01:41,067 - INFO - Created VN __public__, UUID :df24df60-5b33-47bb-af2e-9db596ecf4c1 2026-03-17 15:01:41,400 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-17 15:01:41,420 - DEBUG - Requesting: http://10.0.0.50:8082/domain/e2b92d88-1e3c-43cc-a74b-532eca62d2ba 2026-03-17 15:01:41,458 - DEBUG - Requesting: http://10.0.0.50:8082/project/9f9654e6-9f79-4b99-80a9-d26eacded17b 2026-03-17 15:01:41,660 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/df24df60-5b33-47bb-af2e-9db596ecf4c1 2026-03-17 15:01:41,765 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/df24df60-5b33-47bb-af2e-9db596ecf4c1 2026-03-17 15:01:41,782 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/4282e9d4-91f2-453a-b0a3-9717d36386c6 2026-03-17 15:01:41,849 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/4282e9d4-91f2-453a-b0a3-9717d36386c6 2026-03-17 15:01:41,878 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/48ffe177-038d-4fb3-8a7f-1a0b63cb7b1a 2026-03-17 15:01:41,905 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-17 15:01:41,905 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/df24df60-5b33-47bb-af2e-9db596ecf4c1 2026-03-17 15:01:41,958 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/4282e9d4-91f2-453a-b0a3-9717d36386c6 2026-03-17 15:01:41,999 - INFO - Verified VN network id 13 for VN df24df60-5b33-47bb-af2e-9db596ecf4c1 2026-03-17 15:01:41,999 - INFO - Verifications in API Server for VN __public__ passed 2026-03-17 15:01:42,001 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/df24df60-5b33-47bb-af2e-9db596ecf4c1 2026-03-17 15:01:42,031 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/4282e9d4-91f2-453a-b0a3-9717d36386c6 2026-03-17 15:01:42,042 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/4282e9d4-91f2-453a-b0a3-9717d36386c6 2026-03-17 15:01:42,053 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/48ffe177-038d-4fb3-8a7f-1a0b63cb7b1a 2026-03-17 15:01:42,069 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-17 15:01:42,088 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16079222174559455163', 'uuid-lslong': '12623200208812831937'}, 'enable': 'true', 'created': '2026-03-17T15:01:40', 'last-modified': '2026-03-17T15:01:41', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.800954'} 2026-03-17 15:01:42,089 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-17 15:01:42,135 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-17 15:01:42,135 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-03-17 15:01:42,136 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-03-17 15:01:42,136 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-17 15:01:42,152 - DEBUG - Requesting: http://10.0.0.50:8082/domain/e2b92d88-1e3c-43cc-a74b-532eca62d2ba 2026-03-17 15:01:42,168 - DEBUG - Requesting: http://10.0.0.50:8082/project/9f9654e6-9f79-4b99-80a9-d26eacded17b 2026-03-17 15:01:42,300 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/df24df60-5b33-47bb-af2e-9db596ecf4c1 2026-03-17 15:01:42,320 - DEBUG - =>VN __public__ has no policy to be verified 2026-03-17 15:01:42,320 - DEBUG - Verifying the vn in opserver 2026-03-17 15:01:42,320 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.50 2026-03-17 15:01:42,320 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-03-17 15:01:42,328 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-03-17 15:01:42,329 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-03-17 15:01:42,329 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-17 15:01:42,338 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-17 15:01:42,347 - DEBUG - Do not have enough data to verify VN in agent 2026-03-17 15:01:42,347 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-17 15:01:42,355 - DEBUG - VRF ids for VN __public__: {} 2026-03-17 15:01:42,484 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-03-17 15:01:42,673 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-17 15:01:42,692 - DEBUG - Requesting: http://10.0.0.50:8082/domain/e2b92d88-1e3c-43cc-a74b-532eca62d2ba 2026-03-17 15:01:42,709 - DEBUG - Requesting: http://10.0.0.50:8082/project/9f9654e6-9f79-4b99-80a9-d26eacded17b 2026-03-17 15:01:42,794 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/df24df60-5b33-47bb-af2e-9db596ecf4c1 2026-03-17 15:01:42,826 - DEBUG - Requesting: http://10.0.0.50:8082/floating-ip-pool/bd3057cb-f69b-4442-a664-7d9caf601b4d 2026-03-17 15:01:42,837 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-17 15:01:42,844 - DEBUG - Requesting: http://10.0.0.50:8082/domain/e2b92d88-1e3c-43cc-a74b-532eca62d2ba 2026-03-17 15:01:42,853 - DEBUG - Requesting: http://10.0.0.50:8082/project/9f9654e6-9f79-4b99-80a9-d26eacded17b 2026-03-17 15:01:42,954 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/df24df60-5b33-47bb-af2e-9db596ecf4c1 2026-03-17 15:01:42,981 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-03-17 15:01:42,981 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-03-17 15:01:42,999 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-17 15:01:43,009 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-03-17 15:01:43,009 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-03-17 15:01:43,019 - DEBUG - Ingress ctest-nginx-ingress-45096610 not present 2026-03-17 15:01:43,019 - INFO - Creating Ingress ctest-nginx-ingress-45096610 2026-03-17 15:01:43,033 - INFO - Created Ingress ctest-nginx-ingress-45096610 2026-03-17 15:01:43,040 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-45096610 2026-03-17 15:01:46,051 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-45096610 2026-03-17 15:01:49,061 - DEBUG - For Ingress ctest-nginx-ingress-45096610, Cluster IP: 10.47.255.249, External IPs ['23.50.17.60'] 2026-03-17 15:01:49,061 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-45096610 2026-03-17 15:01:49,061 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=3e0b05c7-4909-418a-8ae8-57f2ce3eef23 2026-03-17 15:01:49,078 - INFO - Ingress ctest-nginx-ingress-45096610 with uuid 3e0b05c7-4909-418a-8ae8-57f2ce3eef23 found in kube manager 2026-03-17 15:01:49,078 - INFO - Ingress ctest-nginx-ingress-45096610 verification passed 2026-03-17 15:01:49,083 - DEBUG - Pod ctest-busybox-pod-86799754 not present 2026-03-17 15:01:49,083 - INFO - Creating Pod ctest-busybox-pod-86799754 2026-03-17 15:01:49,196 - DEBUG - Pod : ctest-busybox-pod-86799754 UUID is ced76333-2d27-4715-8d22-e7c88625b11c 2026-03-17 15:01:49,217 - DEBUG - Pod ctest-nginx-pod-13666412 not in running state.Currently in Pending 2026-03-17 15:01:54,230 - DEBUG - Pod ctest-nginx-pod-13666412 not in running state.Currently in Pending 2026-03-17 15:01:59,244 - INFO - Pod ctest-nginx-pod-13666412 is in running state.Got IP 10.47.255.251 2026-03-17 15:01:59,271 - DEBUG - Pod ctest-nginx-pod-13666412 has vmi 35c668d8-2212-11f1-b329-fa163e846eee 2026-03-17 15:01:59,271 - INFO - Verified pod ctest-nginx-pod-13666412 in contrail-api 2026-03-17 15:01:59,271 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-17 15:01:59,289 - DEBUG - VMI 35c668d8-2212-11f1-b329-fa163e846eee is active in agent 10.0.0.33 2026-03-17 15:01:59,289 - INFO - Verified Pod ctest-nginx-pod-13666412 in agent 10.0.0.33 2026-03-17 15:01:59,290 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=af114e27-b115-4c01-b6ae-10bdf412a1e5 2026-03-17 15:01:59,296 - INFO - Pod ctest-nginx-pod-13666412 with uuid af114e27-b115-4c01-b6ae-10bdf412a1e5 found in kube manager 2026-03-17 15:01:59,296 - INFO - Pod ctest-nginx-pod-13666412 verification passed 2026-03-17 15:01:59,372 - DEBUG - [Pod ctest-nginx-pod-13666412] Cmd: echo ctest-nginx-pod-13666412 > /usr/share/nginx/html/index.html, Output: 2026-03-17 15:01:59,389 - INFO - Pod ctest-nginx-pod-92705561 is in running state.Got IP 10.47.255.250 2026-03-17 15:01:59,464 - DEBUG - Pod ctest-nginx-pod-92705561 has vmi 3668e112-2212-11f1-b329-fa163e846eee 2026-03-17 15:01:59,464 - INFO - Verified pod ctest-nginx-pod-92705561 in contrail-api 2026-03-17 15:01:59,465 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-17 15:01:59,477 - DEBUG - VMI 3668e112-2212-11f1-b329-fa163e846eee is active in agent 10.0.0.33 2026-03-17 15:01:59,477 - INFO - Verified Pod ctest-nginx-pod-92705561 in agent 10.0.0.33 2026-03-17 15:01:59,477 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=8d634122-d636-4a88-b1f3-5e04787b0a30 2026-03-17 15:01:59,485 - INFO - Pod ctest-nginx-pod-92705561 with uuid 8d634122-d636-4a88-b1f3-5e04787b0a30 found in kube manager 2026-03-17 15:01:59,485 - INFO - Pod ctest-nginx-pod-92705561 verification passed 2026-03-17 15:01:59,557 - DEBUG - [Pod ctest-nginx-pod-92705561] Cmd: echo ctest-nginx-pod-92705561 > /usr/share/nginx/html/index.html, Output: 2026-03-17 15:01:59,567 - INFO - Pod ctest-busybox-pod-86799754 is in running state.Got IP 10.47.255.248 2026-03-17 15:01:59,585 - DEBUG - Pod ctest-busybox-pod-86799754 has vmi 39a50108-2212-11f1-b329-fa163e846eee 2026-03-17 15:01:59,585 - INFO - Verified pod ctest-busybox-pod-86799754 in contrail-api 2026-03-17 15:01:59,585 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-17 15:01:59,595 - DEBUG - VMI 39a50108-2212-11f1-b329-fa163e846eee is active in agent 10.0.0.33 2026-03-17 15:01:59,595 - INFO - Verified Pod ctest-busybox-pod-86799754 in agent 10.0.0.33 2026-03-17 15:01:59,596 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=ced76333-2d27-4715-8d22-e7c88625b11c 2026-03-17 15:01:59,601 - INFO - Pod ctest-busybox-pod-86799754 with uuid ced76333-2d27-4715-8d22-e7c88625b11c found in kube manager 2026-03-17 15:01:59,601 - INFO - Pod ctest-busybox-pod-86799754 verification passed 2026-03-17 15:01:59,665 - DEBUG - [Pod ctest-busybox-pod-86799754] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-92705561 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-17 15:01:59,665 - DEBUG - [Pod Pod ctest-busybox-pod-86799754] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-03-17 15:01:59,665 - DEBUG - [Pod Pod ctest-busybox-pod-86799754] Cmd output: ctest-nginx-pod-92705561 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-17 15:01:59,722 - DEBUG - [Pod ctest-busybox-pod-86799754] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-92705561 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-17 15:01:59,722 - DEBUG - [Pod Pod ctest-busybox-pod-86799754] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-03-17 15:01:59,722 - DEBUG - [Pod Pod ctest-busybox-pod-86799754] Cmd output: ctest-nginx-pod-92705561 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-17 15:01:59,782 - DEBUG - [Pod ctest-busybox-pod-86799754] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-92705561 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-17 15:01:59,783 - DEBUG - [Pod Pod ctest-busybox-pod-86799754] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-03-17 15:01:59,783 - DEBUG - [Pod Pod ctest-busybox-pod-86799754] Cmd output: ctest-nginx-pod-92705561 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-17 15:01:59,843 - DEBUG - [Pod ctest-busybox-pod-86799754] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-92705561 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-17 15:01:59,843 - DEBUG - [Pod Pod ctest-busybox-pod-86799754] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-03-17 15:01:59,843 - DEBUG - [Pod Pod ctest-busybox-pod-86799754] Cmd output: ctest-nginx-pod-92705561 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-17 15:01:59,897 - DEBUG - [Pod ctest-busybox-pod-86799754] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-13666412 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-17 15:01:59,897 - DEBUG - [Pod Pod ctest-busybox-pod-86799754] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-03-17 15:01:59,898 - DEBUG - [Pod Pod ctest-busybox-pod-86799754] Cmd output: ctest-nginx-pod-13666412 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-17 15:01:59,898 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-13666412': 1, 'ctest-nginx-pod-92705561': 4} 2026-03-17 15:01:59,898 - WARNING - Nothing to delete parallely 2026-03-17 15:01:59,898 - INFO - Deleting pod default:ctest-busybox-pod-86799754 2026-03-17 15:01:59,916 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=ced76333-2d27-4715-8d22-e7c88625b11c 2026-03-17 15:01:59,925 - WARNING - Pod uuid ced76333-2d27-4715-8d22-e7c88625b11c is still seen in agent 10.0.0.33 VM list 2026-03-17 15:02:04,926 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=ced76333-2d27-4715-8d22-e7c88625b11c 2026-03-17 15:02:04,934 - WARNING - Pod uuid ced76333-2d27-4715-8d22-e7c88625b11c is still seen in agent 10.0.0.33 VM list 2026-03-17 15:02:09,935 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=ced76333-2d27-4715-8d22-e7c88625b11c 2026-03-17 15:02:09,945 - WARNING - Pod uuid ced76333-2d27-4715-8d22-e7c88625b11c is still seen in agent 10.0.0.33 VM list 2026-03-17 15:02:14,945 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=ced76333-2d27-4715-8d22-e7c88625b11c 2026-03-17 15:02:14,957 - WARNING - Pod uuid ced76333-2d27-4715-8d22-e7c88625b11c is still seen in agent 10.0.0.33 VM list 2026-03-17 15:02:19,958 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=ced76333-2d27-4715-8d22-e7c88625b11c 2026-03-17 15:02:19,970 - WARNING - Pod uuid ced76333-2d27-4715-8d22-e7c88625b11c is still seen in agent 10.0.0.33 VM list 2026-03-17 15:02:24,970 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=ced76333-2d27-4715-8d22-e7c88625b11c 2026-03-17 15:02:24,978 - WARNING - Pod uuid ced76333-2d27-4715-8d22-e7c88625b11c is still seen in agent 10.0.0.33 VM list 2026-03-17 15:02:29,980 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=ced76333-2d27-4715-8d22-e7c88625b11c 2026-03-17 15:02:29,993 - WARNING - Pod uuid ced76333-2d27-4715-8d22-e7c88625b11c is still seen in agent 10.0.0.33 VM list 2026-03-17 15:02:34,993 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=ced76333-2d27-4715-8d22-e7c88625b11c 2026-03-17 15:02:35,005 - DEBUG - Pod ced76333-2d27-4715-8d22-e7c88625b11c is not in agent 10.0.0.33 VM list 2026-03-17 15:02:35,006 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:39a50108-2212-11f1-b329-fa163e846eee 2026-03-17 15:02:35,013 - DEBUG - VMI 39a50108-2212-11f1-b329-fa163e846eee is removed from agent 10.0.0.33 2026-03-17 15:02:35,013 - INFO - Verified that pod ctest-busybox-pod-86799754 is removed in agent 2026-03-17 15:02:35,013 - INFO - Deleting Ingress : ctest-nginx-ingress-45096610 2026-03-17 15:02:35,023 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=3e0b05c7-4909-418a-8ae8-57f2ce3eef23 2026-03-17 15:02:35,032 - ERROR - Ingress ctest-nginx-ingress-45096610 with uuid 3e0b05c7-4909-418a-8ae8-57f2ce3eef23 still found in kube manager 2026-03-17 15:02:36,033 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=3e0b05c7-4909-418a-8ae8-57f2ce3eef23 2026-03-17 15:02:36,039 - ERROR - Ingress ctest-nginx-ingress-45096610 with uuid 3e0b05c7-4909-418a-8ae8-57f2ce3eef23 still found in kube manager 2026-03-17 15:02:37,040 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=3e0b05c7-4909-418a-8ae8-57f2ce3eef23 2026-03-17 15:02:37,045 - INFO - Ingress ctest-nginx-ingress-45096610 with uuid 3e0b05c7-4909-418a-8ae8-57f2ce3eef23 deleted successfully from kube manager 2026-03-17 15:02:37,047 - INFO - Verifications on Ingress %s cleanup passed 2026-03-17 15:02:37,047 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-03-17 15:02:37,217 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-17 15:02:37,226 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-03-17 15:02:37,227 - INFO - Deleting VN __public__ 2026-03-17 15:02:37,470 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/4282e9d4-91f2-453a-b0a3-9717d36386c6 2026-03-17 15:02:37,505 - DEBUG - Response Code: 404 2026-03-17 15:02:37,505 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-17 15:02:37,524 - DEBUG - Requesting: http://10.0.0.50:8082/domain/e2b92d88-1e3c-43cc-a74b-532eca62d2ba 2026-03-17 15:02:37,543 - DEBUG - Requesting: http://10.0.0.50:8082/project/9f9654e6-9f79-4b99-80a9-d26eacded17b 2026-03-17 15:02:37,678 - INFO - Validated that VN __public__ is not found in API Server 2026-03-17 15:02:37,678 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-03-17 15:02:37,693 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-17 15:02:37,700 - DEBUG - VN __public__ is not present in Agent 10.0.0.33 2026-03-17 15:02:37,700 - INFO - Validated that VN __public__ is not in any agent 2026-03-17 15:02:37,700 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-17 15:02:37,708 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-17 15:02:37,731 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-17 15:02:37,751 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-03-17 15:02:37,752 - INFO - Deleting pod default:ctest-nginx-pod-92705561 2026-03-17 15:02:37,784 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=8d634122-d636-4a88-b1f3-5e04787b0a30 2026-03-17 15:02:37,803 - WARNING - Pod uuid 8d634122-d636-4a88-b1f3-5e04787b0a30 is still seen in agent 10.0.0.33 VM list 2026-03-17 15:02:42,804 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=8d634122-d636-4a88-b1f3-5e04787b0a30 2026-03-17 15:02:42,812 - DEBUG - Pod 8d634122-d636-4a88-b1f3-5e04787b0a30 is not in agent 10.0.0.33 VM list 2026-03-17 15:02:42,812 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3668e112-2212-11f1-b329-fa163e846eee 2026-03-17 15:02:42,820 - DEBUG - VMI 3668e112-2212-11f1-b329-fa163e846eee is removed from agent 10.0.0.33 2026-03-17 15:02:42,820 - INFO - Verified that pod ctest-nginx-pod-92705561 is removed in agent 2026-03-17 15:02:42,821 - INFO - Deleting pod default:ctest-nginx-pod-13666412 2026-03-17 15:02:42,876 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=af114e27-b115-4c01-b6ae-10bdf412a1e5 2026-03-17 15:02:42,885 - WARNING - Pod uuid af114e27-b115-4c01-b6ae-10bdf412a1e5 is still seen in agent 10.0.0.33 VM list 2026-03-17 15:02:47,886 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=af114e27-b115-4c01-b6ae-10bdf412a1e5 2026-03-17 15:02:47,895 - DEBUG - Pod af114e27-b115-4c01-b6ae-10bdf412a1e5 is not in agent 10.0.0.33 VM list 2026-03-17 15:02:47,896 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:35c668d8-2212-11f1-b329-fa163e846eee 2026-03-17 15:02:47,904 - DEBUG - VMI 35c668d8-2212-11f1-b329-fa163e846eee is removed from agent 10.0.0.33 2026-03-17 15:02:47,905 - INFO - Verified that pod ctest-nginx-pod-13666412 is removed in agent 2026-03-17 15:02:47,905 - INFO - Deleting service : ctest-nginx-svc-03129834 2026-03-17 15:02:48,194 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-17 15:02:48,206 - DEBUG - No XMPP flaps were noticed during the test