2026-01-30 13:29:05,467 - DEBUG - [10.0.0.64]: Running cmd : hostname 2026-01-30 13:29:05,722 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1816-1 2026-01-30 13:29:05,722 - DEBUG - [10.0.0.64]: Running cmd : hostname -f 2026-01-30 13:29:05,755 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1816-1 2026-01-30 13:29:05,756 - DEBUG - [10.0.0.64]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-01-30 13:29:05,809 - DEBUG - Output : NAMES contrail_test_cP8LynirV contrail-kubernetes-master-kubemanager-1 analytics_database-cassandra-1 analytics_database-query-engine-1 analytics_database-nodemgr-1 analytics_snmp-nodemgr-1 analytics_snmp-topology-1 analytics_snmp-snmp-collector-1 analytics_alarm-nodemgr-1 analytics_alarm-alarm-gen-1 analytics_alarm-kafka-1 analytics-api-1 analytics-nodemgr-1 analytics-collector-1 redis-redis-1 webui-web-1 webui-job-1 control-named-1 control-control-1 control-dns-1 control-nodemgr-1 config_database-zookeeper-1 config_database-rabbitmq-1 config_database-nodemgr-1 config_database-cassandra-1 config_api-devicemgr-1 config_api-schema-1 config_api-api-1 config_api-dnsmasq-1 config_api-svcmonitor-1 config_api-nodemgr-1 2026-01-30 13:29:05,809 - DEBUG - [10.0.0.64]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-01-30 13:29:05,894 - DEBUG - Output : 127.0.0.1/8 10.0.0.64/24 252.64.0.1/8 10.126.91.1/24 172.17.0.1/16 2026-01-30 13:29:05,894 - DEBUG - [10.0.0.64]: Running cmd : getent hosts 10.0.0.64 | head -n 1 | awk '{print $2}' 2026-01-30 13:29:05,939 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1816-1 2026-01-30 13:29:06,026 - DEBUG - [10.0.0.72]: Running cmd : hostname 2026-01-30 13:29:06,267 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1816-1 2026-01-30 13:29:06,267 - DEBUG - [10.0.0.72]: Running cmd : hostname -f 2026-01-30 13:29:06,291 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1816-1 2026-01-30 13:29:06,292 - DEBUG - [10.0.0.72]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-01-30 13:29:06,380 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2026-01-30 13:29:06,380 - DEBUG - [10.0.0.72]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-01-30 13:29:06,414 - DEBUG - Output : 10.0.0.72/24 2026-01-30 13:29:06,414 - DEBUG - [10.0.0.72]: Running cmd : getent hosts 10.0.0.72 | head -n 1 | awk '{print $2}' 2026-01-30 13:29:06,449 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1816-1 2026-01-30 13:29:06,521 - INFO - Using existing project ['default-domain', 'k8s-default'](c842824e-bb35-4cc4-9755-1b0edba2f678) 2026-01-30 13:29:06,635 - INFO - Using existing project ['default-domain', 'k8s-default'](c842824e-bb35-4cc4-9755-1b0edba2f678) 2026-01-30 13:29:06,744 - INFO - ================================================================================ 2026-01-30 13:29:06,745 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-01-30 13:29:06,745 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-01-30 13:29:06,963 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_AgentXmppConnectionStatusReq? 2026-01-30 13:29:06,971 - DEBUG - Nothing to compare xmpp stats {'10.0.0.72': {'10.0.0.64': '0'}} with 2026-01-30 13:29:06,971 - INFO - Initial checks done. Running the testcase now 2026-01-30 13:29:06,972 - INFO - 2026-01-30 13:29:06,984 - INFO - Namespace default already exists 2026-01-30 13:29:06,985 - DEBUG - [10.0.0.64]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-01-30 13:29:07,110 - DEBUG - Output : cluster_project={} 2026-01-30 13:29:07,144 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-01-30 13:29:07,144 - DEBUG - Project uuid in Contrail is c842824e-bb35-4cc4-9755-1b0edba2f678 2026-01-30 13:29:07,144 - DEBUG - [10.0.0.64]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-01-30 13:29:07,261 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-01-30 13:29:07,262 - DEBUG - Requesting: http://10.0.0.64:8108/Snh_NamespaceDatabaseList?namespace_uuid=22601ebe-e344-4f3d-a750-09de352e8ecb 2026-01-30 13:29:07,265 - INFO - Namespace default with uuid 22601ebe-e344-4f3d-a750-09de352e8ecb found in kube manager 2026-01-30 13:29:07,265 - INFO - Namespace default verification passed 2026-01-30 13:29:07,268 - DEBUG - Service ctest-nginx-svc-05126183 not present 2026-01-30 13:29:07,268 - INFO - Creating service ctest-nginx-svc-05126183 2026-01-30 13:29:07,288 - INFO - Created Service ctest-nginx-svc-05126183 2026-01-30 13:29:07,294 - DEBUG - Pod ctest-nginx-pod-00717798 not present 2026-01-30 13:29:07,295 - INFO - Creating Pod ctest-nginx-pod-00717798 2026-01-30 13:29:07,364 - DEBUG - Pod : ctest-nginx-pod-00717798 UUID is abe89b98-7ab5-449f-9638-3c2b5f07ff2a 2026-01-30 13:29:07,367 - DEBUG - Pod ctest-nginx-pod-35713308 not present 2026-01-30 13:29:07,368 - INFO - Creating Pod ctest-nginx-pod-35713308 2026-01-30 13:29:07,394 - DEBUG - Pod : ctest-nginx-pod-35713308 UUID is 1c06835e-2a83-49b2-9cd2-62f3a4e53307 2026-01-30 13:29:07,532 - INFO - Created VN __public__, UUID :c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:07,715 - DEBUG - Requesting: http://10.0.0.64:8082/domains 2026-01-30 13:29:07,723 - DEBUG - Requesting: http://10.0.0.64:8082/domain/658a0981-017a-4986-b36a-203aa53777c1 2026-01-30 13:29:07,745 - DEBUG - Requesting: http://10.0.0.64:8082/project/c842824e-bb35-4cc4-9755-1b0edba2f678 2026-01-30 13:29:07,800 - DEBUG - Requesting: http://10.0.0.64:8082/virtual-network/c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:07,812 - DEBUG - Requesting: http://10.0.0.64:8082/virtual-network/c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:07,821 - DEBUG - Requesting: http://10.0.0.64:8082/routing-instance/9ba4a795-3465-4b6d-9f81-dd9328dbdb3e 2026-01-30 13:29:07,829 - DEBUG - Requesting: http://10.0.0.64:8082/routing-instance/9ba4a795-3465-4b6d-9f81-dd9328dbdb3e 2026-01-30 13:29:07,837 - DEBUG - Route Targets: [] 2026-01-30 13:29:07,838 - DEBUG - RT names not yet present for VN __public__ 2026-01-30 13:29:12,839 - DEBUG - Requesting: http://10.0.0.64:8082/domains 2026-01-30 13:29:12,844 - DEBUG - Requesting: http://10.0.0.64:8082/domain/658a0981-017a-4986-b36a-203aa53777c1 2026-01-30 13:29:12,851 - DEBUG - Requesting: http://10.0.0.64:8082/project/c842824e-bb35-4cc4-9755-1b0edba2f678 2026-01-30 13:29:12,896 - DEBUG - Requesting: http://10.0.0.64:8082/virtual-network/c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:12,904 - DEBUG - Requesting: http://10.0.0.64:8082/virtual-network/c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:12,914 - DEBUG - Requesting: http://10.0.0.64:8082/routing-instance/9ba4a795-3465-4b6d-9f81-dd9328dbdb3e 2026-01-30 13:29:12,921 - DEBUG - Requesting: http://10.0.0.64:8082/routing-instance/9ba4a795-3465-4b6d-9f81-dd9328dbdb3e 2026-01-30 13:29:12,927 - DEBUG - Requesting: http://10.0.0.64:8082/route-target/95b3c98f-d9e8-4f9c-880b-b38bf7554507 2026-01-30 13:29:12,931 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-01-30 13:29:12,931 - DEBUG - Requesting: http://10.0.0.64:8082/virtual-network/c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:12,938 - DEBUG - Requesting: http://10.0.0.64:8082/routing-instance/9ba4a795-3465-4b6d-9f81-dd9328dbdb3e 2026-01-30 13:29:12,950 - INFO - Verified VN network id 13 for VN c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:12,950 - INFO - Verifications in API Server for VN __public__ passed 2026-01-30 13:29:12,951 - DEBUG - Requesting: http://10.0.0.64:8082/virtual-network/c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:12,958 - DEBUG - Requesting: http://10.0.0.64:8082/routing-instance/9ba4a795-3465-4b6d-9f81-dd9328dbdb3e 2026-01-30 13:29:12,964 - DEBUG - Requesting: http://10.0.0.64:8082/routing-instance/9ba4a795-3465-4b6d-9f81-dd9328dbdb3e 2026-01-30 13:29:12,968 - DEBUG - Requesting: http://10.0.0.64:8082/route-target/95b3c98f-d9e8-4f9c-880b-b38bf7554507 2026-01-30 13:29:12,973 - DEBUG - Requesting: http://10.0.0.64:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-01-30 13:29:12,979 - DEBUG - Control-node 10.0.0.64 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13916139033591760362', 'uuid-lslong': '9646337104977016323'}, 'enable': 'true', 'created': '2026-01-30T13:29:07', 'last-modified': '2026-01-30T13:29:07', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.350203'} 2026-01-30 13:29:12,979 - DEBUG - Requesting: http://10.0.0.64:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 13:29:12,989 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-01-30 13:29:12,989 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-01-30 13:29:12,990 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-01-30 13:29:12,990 - DEBUG - Requesting: http://10.0.0.64:8082/domains 2026-01-30 13:29:12,995 - DEBUG - Requesting: http://10.0.0.64:8082/domain/658a0981-017a-4986-b36a-203aa53777c1 2026-01-30 13:29:13,006 - DEBUG - Requesting: http://10.0.0.64:8082/project/c842824e-bb35-4cc4-9755-1b0edba2f678 2026-01-30 13:29:13,042 - DEBUG - Requesting: http://10.0.0.64:8082/virtual-network/c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:13,051 - DEBUG - =>VN __public__ has no policy to be verified 2026-01-30 13:29:13,051 - DEBUG - Verifying the vn in opserver 2026-01-30 13:29:13,051 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.64 2026-01-30 13:29:13,051 - DEBUG - Requesting: http://10.0.0.64:8081/analytics/uves/virtual-networks 2026-01-30 13:29:13,055 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.64:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-01-30 13:29:13,055 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-01-30 13:29:13,056 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 13:29:13,063 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 13:29:13,068 - DEBUG - Do not have enough data to verify VN in agent 2026-01-30 13:29:13,068 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 13:29:13,072 - DEBUG - VRF ids for VN __public__: {} 2026-01-30 13:29:13,167 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-01-30 13:29:13,262 - DEBUG - Requesting: http://10.0.0.64:8082/domains 2026-01-30 13:29:13,266 - DEBUG - Requesting: http://10.0.0.64:8082/domain/658a0981-017a-4986-b36a-203aa53777c1 2026-01-30 13:29:13,271 - DEBUG - Requesting: http://10.0.0.64:8082/project/c842824e-bb35-4cc4-9755-1b0edba2f678 2026-01-30 13:29:13,308 - DEBUG - Requesting: http://10.0.0.64:8082/virtual-network/c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:13,326 - DEBUG - Requesting: http://10.0.0.64:8082/floating-ip-pool/f2d34635-77e5-4ab3-9a63-62d93248f36a 2026-01-30 13:29:13,336 - DEBUG - Requesting: http://10.0.0.64:8082/domains 2026-01-30 13:29:13,347 - DEBUG - Requesting: http://10.0.0.64:8082/domain/658a0981-017a-4986-b36a-203aa53777c1 2026-01-30 13:29:13,356 - DEBUG - Requesting: http://10.0.0.64:8082/project/c842824e-bb35-4cc4-9755-1b0edba2f678 2026-01-30 13:29:13,385 - DEBUG - Requesting: http://10.0.0.64:8082/virtual-network/c1200eb8-5e2f-45ea-85de-ac7d0dd33603 2026-01-30 13:29:13,399 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-01-30 13:29:13,399 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-01-30 13:29:13,407 - DEBUG - Requesting: http://10.0.0.64:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-01-30 13:29:13,413 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-01-30 13:29:13,413 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-01-30 13:29:13,416 - DEBUG - Ingress ctest-nginx-ingress-33815858 not present 2026-01-30 13:29:13,417 - INFO - Creating Ingress ctest-nginx-ingress-33815858 2026-01-30 13:29:13,433 - INFO - Created Ingress ctest-nginx-ingress-33815858 2026-01-30 13:29:13,437 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-33815858 2026-01-30 13:29:16,444 - DEBUG - For Ingress ctest-nginx-ingress-33815858, Cluster IP: 10.47.255.249, External IPs ['11.63.106.60'] 2026-01-30 13:29:16,444 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-33815858 2026-01-30 13:29:16,444 - DEBUG - Requesting: http://10.0.0.64:8108/Snh_LoadbalancerDatabaseList?x=fc1faf57-070c-4726-a682-39c29275fb77 2026-01-30 13:29:16,449 - INFO - Ingress ctest-nginx-ingress-33815858 with uuid fc1faf57-070c-4726-a682-39c29275fb77 found in kube manager 2026-01-30 13:29:16,449 - INFO - Ingress ctest-nginx-ingress-33815858 verification passed 2026-01-30 13:29:16,452 - DEBUG - Pod ctest-busybox-pod-07715120 not present 2026-01-30 13:29:16,452 - INFO - Creating Pod ctest-busybox-pod-07715120 2026-01-30 13:29:16,483 - DEBUG - Pod : ctest-busybox-pod-07715120 UUID is 6ab5adb7-0fd2-48c5-afc5-675b52c0767f 2026-01-30 13:29:16,494 - DEBUG - Pod ctest-nginx-pod-00717798 not in running state.Currently in Pending 2026-01-30 13:29:21,501 - INFO - Pod ctest-nginx-pod-00717798 is in running state.Got IP 10.47.255.251 2026-01-30 13:29:21,512 - DEBUG - Pod ctest-nginx-pod-00717798 has vmi a7fa6c30-fddf-11f0-9bea-fa163ec1d9df 2026-01-30 13:29:21,513 - INFO - Verified pod ctest-nginx-pod-00717798 in contrail-api 2026-01-30 13:29:21,513 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-01-30 13:29:21,523 - DEBUG - VMI a7fa6c30-fddf-11f0-9bea-fa163ec1d9df is active in agent 10.0.0.72 2026-01-30 13:29:21,523 - INFO - Verified Pod ctest-nginx-pod-00717798 in agent 10.0.0.72 2026-01-30 13:29:21,523 - DEBUG - Requesting: http://10.0.0.64:8108/Snh_VirtualMachineDatabaseList?x=abe89b98-7ab5-449f-9638-3c2b5f07ff2a 2026-01-30 13:29:21,527 - INFO - Pod ctest-nginx-pod-00717798 with uuid abe89b98-7ab5-449f-9638-3c2b5f07ff2a found in kube manager 2026-01-30 13:29:21,527 - INFO - Pod ctest-nginx-pod-00717798 verification passed 2026-01-30 13:29:21,591 - DEBUG - [Pod ctest-nginx-pod-00717798] Cmd: echo ctest-nginx-pod-00717798 > /usr/share/nginx/html/index.html, Output: 2026-01-30 13:29:21,598 - DEBUG - Pod ctest-nginx-pod-35713308 not in running state.Currently in Pending 2026-01-30 13:29:26,612 - INFO - Pod ctest-nginx-pod-35713308 is in running state.Got IP 10.47.255.250 2026-01-30 13:29:26,667 - DEBUG - Pod ctest-nginx-pod-35713308 has vmi a8658916-fddf-11f0-9bea-fa163ec1d9df 2026-01-30 13:29:26,667 - INFO - Verified pod ctest-nginx-pod-35713308 in contrail-api 2026-01-30 13:29:26,667 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-01-30 13:29:26,677 - DEBUG - VMI a8658916-fddf-11f0-9bea-fa163ec1d9df is active in agent 10.0.0.72 2026-01-30 13:29:26,677 - INFO - Verified Pod ctest-nginx-pod-35713308 in agent 10.0.0.72 2026-01-30 13:29:26,677 - DEBUG - Requesting: http://10.0.0.64:8108/Snh_VirtualMachineDatabaseList?x=1c06835e-2a83-49b2-9cd2-62f3a4e53307 2026-01-30 13:29:26,683 - INFO - Pod ctest-nginx-pod-35713308 with uuid 1c06835e-2a83-49b2-9cd2-62f3a4e53307 found in kube manager 2026-01-30 13:29:26,683 - INFO - Pod ctest-nginx-pod-35713308 verification passed 2026-01-30 13:29:26,758 - DEBUG - [Pod ctest-nginx-pod-35713308] Cmd: echo ctest-nginx-pod-35713308 > /usr/share/nginx/html/index.html, Output: 2026-01-30 13:29:26,765 - DEBUG - Pod ctest-busybox-pod-07715120 not in running state.Currently in Pending 2026-01-30 13:29:31,773 - INFO - Pod ctest-busybox-pod-07715120 is in running state.Got IP 10.47.255.248 2026-01-30 13:29:31,784 - DEBUG - Pod ctest-busybox-pod-07715120 has vmi acc8d5ee-fddf-11f0-9bea-fa163ec1d9df 2026-01-30 13:29:31,784 - INFO - Verified pod ctest-busybox-pod-07715120 in contrail-api 2026-01-30 13:29:31,784 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-01-30 13:29:31,795 - DEBUG - VMI acc8d5ee-fddf-11f0-9bea-fa163ec1d9df is active in agent 10.0.0.72 2026-01-30 13:29:31,795 - INFO - Verified Pod ctest-busybox-pod-07715120 in agent 10.0.0.72 2026-01-30 13:29:31,796 - DEBUG - Requesting: http://10.0.0.64:8108/Snh_VirtualMachineDatabaseList?x=6ab5adb7-0fd2-48c5-afc5-675b52c0767f 2026-01-30 13:29:31,799 - INFO - Pod ctest-busybox-pod-07715120 with uuid 6ab5adb7-0fd2-48c5-afc5-675b52c0767f found in kube manager 2026-01-30 13:29:31,799 - INFO - Pod ctest-busybox-pod-07715120 verification passed 2026-01-30 13:29:31,855 - DEBUG - [Pod ctest-busybox-pod-07715120] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-00717798 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-01-30 13:29:31,855 - DEBUG - [Pod Pod ctest-busybox-pod-07715120] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-01-30 13:29:31,855 - DEBUG - [Pod Pod ctest-busybox-pod-07715120] Cmd output: ctest-nginx-pod-00717798 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-01-30 13:29:31,903 - DEBUG - [Pod ctest-busybox-pod-07715120] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-35713308 2026-01-30 13:29:31,903 - DEBUG - [Pod Pod ctest-busybox-pod-07715120] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-01-30 13:29:31,903 - DEBUG - [Pod Pod ctest-busybox-pod-07715120] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-35713308 2026-01-30 13:29:31,903 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-00717798': 1, 'ctest-nginx-pod-35713308': 1} 2026-01-30 13:29:31,903 - WARNING - Nothing to delete parallely 2026-01-30 13:29:31,903 - INFO - Deleting pod default:ctest-busybox-pod-07715120 2026-01-30 13:29:31,914 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=6ab5adb7-0fd2-48c5-afc5-675b52c0767f 2026-01-30 13:29:31,926 - WARNING - Pod uuid 6ab5adb7-0fd2-48c5-afc5-675b52c0767f is still seen in agent 10.0.0.72 VM list 2026-01-30 13:29:36,927 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=6ab5adb7-0fd2-48c5-afc5-675b52c0767f 2026-01-30 13:29:36,933 - WARNING - Pod uuid 6ab5adb7-0fd2-48c5-afc5-675b52c0767f is still seen in agent 10.0.0.72 VM list 2026-01-30 13:29:41,934 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=6ab5adb7-0fd2-48c5-afc5-675b52c0767f 2026-01-30 13:29:41,941 - WARNING - Pod uuid 6ab5adb7-0fd2-48c5-afc5-675b52c0767f is still seen in agent 10.0.0.72 VM list 2026-01-30 13:29:46,942 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=6ab5adb7-0fd2-48c5-afc5-675b52c0767f 2026-01-30 13:29:46,948 - WARNING - Pod uuid 6ab5adb7-0fd2-48c5-afc5-675b52c0767f is still seen in agent 10.0.0.72 VM list 2026-01-30 13:29:51,949 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=6ab5adb7-0fd2-48c5-afc5-675b52c0767f 2026-01-30 13:29:51,963 - WARNING - Pod uuid 6ab5adb7-0fd2-48c5-afc5-675b52c0767f is still seen in agent 10.0.0.72 VM list 2026-01-30 13:29:56,963 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=6ab5adb7-0fd2-48c5-afc5-675b52c0767f 2026-01-30 13:29:56,970 - WARNING - Pod uuid 6ab5adb7-0fd2-48c5-afc5-675b52c0767f is still seen in agent 10.0.0.72 VM list 2026-01-30 13:30:01,971 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=6ab5adb7-0fd2-48c5-afc5-675b52c0767f 2026-01-30 13:30:01,978 - WARNING - Pod uuid 6ab5adb7-0fd2-48c5-afc5-675b52c0767f is still seen in agent 10.0.0.72 VM list 2026-01-30 13:30:06,979 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=6ab5adb7-0fd2-48c5-afc5-675b52c0767f 2026-01-30 13:30:06,986 - DEBUG - Pod 6ab5adb7-0fd2-48c5-afc5-675b52c0767f is not in agent 10.0.0.72 VM list 2026-01-30 13:30:06,986 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:acc8d5ee-fddf-11f0-9bea-fa163ec1d9df 2026-01-30 13:30:06,992 - DEBUG - VMI acc8d5ee-fddf-11f0-9bea-fa163ec1d9df is removed from agent 10.0.0.72 2026-01-30 13:30:06,992 - INFO - Verified that pod ctest-busybox-pod-07715120 is removed in agent 2026-01-30 13:30:06,992 - INFO - Deleting Ingress : ctest-nginx-ingress-33815858 2026-01-30 13:30:06,998 - DEBUG - Requesting: http://10.0.0.64:8108/Snh_LoadbalancerDatabaseList?x=fc1faf57-070c-4726-a682-39c29275fb77 2026-01-30 13:30:07,005 - ERROR - Ingress ctest-nginx-ingress-33815858 with uuid fc1faf57-070c-4726-a682-39c29275fb77 still found in kube manager 2026-01-30 13:30:08,007 - DEBUG - Requesting: http://10.0.0.64:8108/Snh_LoadbalancerDatabaseList?x=fc1faf57-070c-4726-a682-39c29275fb77 2026-01-30 13:30:08,010 - INFO - Ingress ctest-nginx-ingress-33815858 with uuid fc1faf57-070c-4726-a682-39c29275fb77 deleted successfully from kube manager 2026-01-30 13:30:08,010 - INFO - Verifications on Ingress %s cleanup passed 2026-01-30 13:30:08,010 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-01-30 13:30:08,102 - DEBUG - Requesting: http://10.0.0.64:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-01-30 13:30:08,106 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-01-30 13:30:08,106 - INFO - Deleting VN __public__ 2026-01-30 13:30:08,209 - DEBUG - Requesting: http://10.0.0.64:8082/routing-instance/9ba4a795-3465-4b6d-9f81-dd9328dbdb3e 2026-01-30 13:30:08,217 - DEBUG - Response Code: 404 2026-01-30 13:30:08,217 - DEBUG - Requesting: http://10.0.0.64:8082/domains 2026-01-30 13:30:08,222 - DEBUG - Requesting: http://10.0.0.64:8082/domain/658a0981-017a-4986-b36a-203aa53777c1 2026-01-30 13:30:08,228 - DEBUG - Requesting: http://10.0.0.64:8082/project/c842824e-bb35-4cc4-9755-1b0edba2f678 2026-01-30 13:30:08,257 - INFO - Validated that VN __public__ is not found in API Server 2026-01-30 13:30:08,257 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-01-30 13:30:08,264 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 13:30:08,270 - DEBUG - VN __public__ is not present in Agent 10.0.0.72 2026-01-30 13:30:08,270 - INFO - Validated that VN __public__ is not in any agent 2026-01-30 13:30:08,270 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 13:30:08,276 - DEBUG - Requesting: http://10.0.0.64:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 13:30:08,280 - DEBUG - Requesting: http://10.0.0.64:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-01-30 13:30:08,284 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-01-30 13:30:08,285 - INFO - Deleting pod default:ctest-nginx-pod-35713308 2026-01-30 13:30:08,308 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=1c06835e-2a83-49b2-9cd2-62f3a4e53307 2026-01-30 13:30:08,332 - WARNING - Pod uuid 1c06835e-2a83-49b2-9cd2-62f3a4e53307 is still seen in agent 10.0.0.72 VM list 2026-01-30 13:30:13,333 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=1c06835e-2a83-49b2-9cd2-62f3a4e53307 2026-01-30 13:30:13,339 - DEBUG - Pod 1c06835e-2a83-49b2-9cd2-62f3a4e53307 is not in agent 10.0.0.72 VM list 2026-01-30 13:30:13,339 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a8658916-fddf-11f0-9bea-fa163ec1d9df 2026-01-30 13:30:13,344 - DEBUG - VMI a8658916-fddf-11f0-9bea-fa163ec1d9df is removed from agent 10.0.0.72 2026-01-30 13:30:13,344 - INFO - Verified that pod ctest-nginx-pod-35713308 is removed in agent 2026-01-30 13:30:13,344 - INFO - Deleting pod default:ctest-nginx-pod-00717798 2026-01-30 13:30:13,357 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=abe89b98-7ab5-449f-9638-3c2b5f07ff2a 2026-01-30 13:30:13,365 - WARNING - Pod uuid abe89b98-7ab5-449f-9638-3c2b5f07ff2a is still seen in agent 10.0.0.72 VM list 2026-01-30 13:30:18,365 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_VmListReq?uuid=abe89b98-7ab5-449f-9638-3c2b5f07ff2a 2026-01-30 13:30:18,372 - DEBUG - Pod abe89b98-7ab5-449f-9638-3c2b5f07ff2a is not in agent 10.0.0.72 VM list 2026-01-30 13:30:18,372 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a7fa6c30-fddf-11f0-9bea-fa163ec1d9df 2026-01-30 13:30:18,377 - DEBUG - VMI a7fa6c30-fddf-11f0-9bea-fa163ec1d9df is removed from agent 10.0.0.72 2026-01-30 13:30:18,377 - INFO - Verified that pod ctest-nginx-pod-00717798 is removed in agent 2026-01-30 13:30:18,378 - INFO - Deleting service : ctest-nginx-svc-05126183 2026-01-30 13:30:18,575 - DEBUG - Requesting: http://10.0.0.72:8085/Snh_AgentXmppConnectionStatusReq? 2026-01-30 13:30:18,581 - DEBUG - No XMPP flaps were noticed during the test