2026-01-30 14:38:15,990 - DEBUG - [10.0.0.131]: Running cmd : hostname 2026-01-30 14:38:16,249 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1818-1 2026-01-30 14:38:16,250 - DEBUG - [10.0.0.131]: Running cmd : hostname -f 2026-01-30 14:38:16,283 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1818-1 2026-01-30 14:38:16,283 - DEBUG - [10.0.0.131]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-01-30 14:38:16,386 - DEBUG - Output : NAMES contrail_test_mQtLos8Kh contrail-kubernetes-master-kubemanager-1 analytics_database-nodemgr-1 analytics_database-query-engine-1 analytics_database-cassandra-1 analytics_snmp-nodemgr-1 analytics_snmp-snmp-collector-1 analytics_snmp-topology-1 analytics_alarm-nodemgr-1 analytics_alarm-alarm-gen-1 analytics_alarm-kafka-1 analytics-api-1 analytics-nodemgr-1 analytics-collector-1 redis-redis-1 webui-web-1 webui-job-1 control-named-1 control-control-1 control-dns-1 control-nodemgr-1 config_database-zookeeper-1 config_database-rabbitmq-1 config_database-cassandra-1 config_database-nodemgr-1 config_api-devicemgr-1 config_api-api-1 config_api-schema-1 config_api-svcmonitor-1 config_api-nodemgr-1 config_api-dnsmasq-1 2026-01-30 14:38:16,387 - DEBUG - [10.0.0.131]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-01-30 14:38:16,421 - DEBUG - Output : 127.0.0.1/8 10.0.0.131/24 252.131.0.1/8 10.49.223.1/24 172.17.0.1/16 2026-01-30 14:38:16,421 - DEBUG - [10.0.0.131]: Running cmd : getent hosts 10.0.0.131 | head -n 1 | awk '{print $2}' 2026-01-30 14:38:16,465 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1818-1 2026-01-30 14:38:16,570 - DEBUG - [10.0.0.136]: Running cmd : hostname 2026-01-30 14:38:16,834 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1818-1 2026-01-30 14:38:16,834 - DEBUG - [10.0.0.136]: Running cmd : hostname -f 2026-01-30 14:38:16,877 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1818-1 2026-01-30 14:38:16,879 - DEBUG - [10.0.0.136]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-01-30 14:38:16,941 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2026-01-30 14:38:16,941 - DEBUG - [10.0.0.136]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-01-30 14:38:17,016 - DEBUG - Output : 10.0.0.136/24 2026-01-30 14:38:17,017 - DEBUG - [10.0.0.136]: Running cmd : getent hosts 10.0.0.136 | head -n 1 | awk '{print $2}' 2026-01-30 14:38:17,051 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1818-1 2026-01-30 14:38:17,134 - INFO - Using existing project ['default-domain', 'k8s-default'](98447ac1-24d6-478e-adcc-dc791d752844) 2026-01-30 14:38:17,243 - INFO - Using existing project ['default-domain', 'k8s-default'](98447ac1-24d6-478e-adcc-dc791d752844) 2026-01-30 14:38:17,349 - INFO - ================================================================================ 2026-01-30 14:38:17,349 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-01-30 14:38:17,349 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-01-30 14:38:17,600 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_AgentXmppConnectionStatusReq? 2026-01-30 14:38:17,608 - DEBUG - Nothing to compare xmpp stats {'10.0.0.136': {'10.0.0.131': '0'}} with 2026-01-30 14:38:17,608 - INFO - Initial checks done. Running the testcase now 2026-01-30 14:38:17,608 - INFO - 2026-01-30 14:38:17,612 - INFO - Namespace default already exists 2026-01-30 14:38:17,613 - DEBUG - [10.0.0.131]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-01-30 14:38:17,729 - DEBUG - Output : cluster_project={} 2026-01-30 14:38:17,773 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-01-30 14:38:17,774 - DEBUG - Project uuid in Contrail is 98447ac1-24d6-478e-adcc-dc791d752844 2026-01-30 14:38:17,774 - DEBUG - [10.0.0.131]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-01-30 14:38:17,893 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-01-30 14:38:17,893 - DEBUG - Requesting: http://10.0.0.131:8108/Snh_NamespaceDatabaseList?namespace_uuid=1dc94a18-1c24-4727-a1b6-e197d86a51e6 2026-01-30 14:38:17,897 - INFO - Namespace default with uuid 1dc94a18-1c24-4727-a1b6-e197d86a51e6 found in kube manager 2026-01-30 14:38:17,897 - INFO - Namespace default verification passed 2026-01-30 14:38:17,900 - DEBUG - Service ctest-nginx-svc-37840724 not present 2026-01-30 14:38:17,901 - INFO - Creating service ctest-nginx-svc-37840724 2026-01-30 14:38:17,917 - INFO - Created Service ctest-nginx-svc-37840724 2026-01-30 14:38:17,923 - DEBUG - Pod ctest-nginx-pod-75763037 not present 2026-01-30 14:38:17,930 - INFO - Creating Pod ctest-nginx-pod-75763037 2026-01-30 14:38:17,991 - DEBUG - Pod : ctest-nginx-pod-75763037 UUID is 0367a583-8c19-4c28-80d1-a42d9e3ec986 2026-01-30 14:38:17,995 - DEBUG - Pod ctest-nginx-pod-59542138 not present 2026-01-30 14:38:17,996 - INFO - Creating Pod ctest-nginx-pod-59542138 2026-01-30 14:38:18,021 - DEBUG - Pod : ctest-nginx-pod-59542138 UUID is ba2705ef-233a-4d67-9f2b-8a9145b0a336 2026-01-30 14:38:18,173 - INFO - Created VN __public__, UUID :fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:18,362 - DEBUG - Requesting: http://10.0.0.131:8082/domains 2026-01-30 14:38:18,369 - DEBUG - Requesting: http://10.0.0.131:8082/domain/bda8e106-cb10-4d85-94cf-36ec65254e20 2026-01-30 14:38:18,390 - DEBUG - Requesting: http://10.0.0.131:8082/project/98447ac1-24d6-478e-adcc-dc791d752844 2026-01-30 14:38:18,451 - DEBUG - Requesting: http://10.0.0.131:8082/virtual-network/fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:18,461 - DEBUG - Requesting: http://10.0.0.131:8082/virtual-network/fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:18,468 - DEBUG - Requesting: http://10.0.0.131:8082/routing-instance/bfed1a7d-5b6b-45b7-b0bd-461c9e8945f4 2026-01-30 14:38:18,475 - DEBUG - Requesting: http://10.0.0.131:8082/routing-instance/bfed1a7d-5b6b-45b7-b0bd-461c9e8945f4 2026-01-30 14:38:18,482 - DEBUG - Route Targets: [] 2026-01-30 14:38:18,482 - DEBUG - RT names not yet present for VN __public__ 2026-01-30 14:38:23,482 - DEBUG - Requesting: http://10.0.0.131:8082/domains 2026-01-30 14:38:23,488 - DEBUG - Requesting: http://10.0.0.131:8082/domain/bda8e106-cb10-4d85-94cf-36ec65254e20 2026-01-30 14:38:23,494 - DEBUG - Requesting: http://10.0.0.131:8082/project/98447ac1-24d6-478e-adcc-dc791d752844 2026-01-30 14:38:23,533 - DEBUG - Requesting: http://10.0.0.131:8082/virtual-network/fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:23,540 - DEBUG - Requesting: http://10.0.0.131:8082/virtual-network/fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:23,548 - DEBUG - Requesting: http://10.0.0.131:8082/routing-instance/bfed1a7d-5b6b-45b7-b0bd-461c9e8945f4 2026-01-30 14:38:23,554 - DEBUG - Requesting: http://10.0.0.131:8082/routing-instance/bfed1a7d-5b6b-45b7-b0bd-461c9e8945f4 2026-01-30 14:38:23,559 - DEBUG - Requesting: http://10.0.0.131:8082/route-target/bd22334d-5586-43eb-9414-a3af717a695a 2026-01-30 14:38:23,564 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-01-30 14:38:23,564 - DEBUG - Requesting: http://10.0.0.131:8082/virtual-network/fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:23,571 - DEBUG - Requesting: http://10.0.0.131:8082/routing-instance/bfed1a7d-5b6b-45b7-b0bd-461c9e8945f4 2026-01-30 14:38:23,581 - INFO - Verified VN network id 13 for VN fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:23,581 - INFO - Verifications in API Server for VN __public__ passed 2026-01-30 14:38:23,581 - DEBUG - Requesting: http://10.0.0.131:8082/virtual-network/fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:23,588 - DEBUG - Requesting: http://10.0.0.131:8082/routing-instance/bfed1a7d-5b6b-45b7-b0bd-461c9e8945f4 2026-01-30 14:38:23,593 - DEBUG - Requesting: http://10.0.0.131:8082/routing-instance/bfed1a7d-5b6b-45b7-b0bd-461c9e8945f4 2026-01-30 14:38:23,598 - DEBUG - Requesting: http://10.0.0.131:8082/route-target/bd22334d-5586-43eb-9414-a3af717a695a 2026-01-30 14:38:23,603 - DEBUG - Requesting: http://10.0.0.131:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-01-30 14:38:23,610 - DEBUG - Control-node 10.0.0.131 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '18280603568646410186', 'uuid-lslong': '12316498084994415575'}, 'enable': 'true', 'created': '2026-01-30T14:38:18', 'last-modified': '2026-01-30T14:38:18', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.313805'} 2026-01-30 14:38:23,610 - DEBUG - Requesting: http://10.0.0.131:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 14:38:23,619 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-01-30 14:38:23,619 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-01-30 14:38:23,619 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-01-30 14:38:23,619 - DEBUG - Requesting: http://10.0.0.131:8082/domains 2026-01-30 14:38:23,626 - DEBUG - Requesting: http://10.0.0.131:8082/domain/bda8e106-cb10-4d85-94cf-36ec65254e20 2026-01-30 14:38:23,634 - DEBUG - Requesting: http://10.0.0.131:8082/project/98447ac1-24d6-478e-adcc-dc791d752844 2026-01-30 14:38:23,672 - DEBUG - Requesting: http://10.0.0.131:8082/virtual-network/fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:23,679 - DEBUG - =>VN __public__ has no policy to be verified 2026-01-30 14:38:23,679 - DEBUG - Verifying the vn in opserver 2026-01-30 14:38:23,679 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.131 2026-01-30 14:38:23,680 - DEBUG - Requesting: http://10.0.0.131:8081/analytics/uves/virtual-networks 2026-01-30 14:38:23,684 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.131:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-01-30 14:38:23,684 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-01-30 14:38:23,685 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 14:38:23,694 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 14:38:23,701 - DEBUG - Do not have enough data to verify VN in agent 2026-01-30 14:38:23,701 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 14:38:23,707 - DEBUG - VRF ids for VN __public__: {} 2026-01-30 14:38:23,806 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-01-30 14:38:23,906 - DEBUG - Requesting: http://10.0.0.131:8082/domains 2026-01-30 14:38:23,911 - DEBUG - Requesting: http://10.0.0.131:8082/domain/bda8e106-cb10-4d85-94cf-36ec65254e20 2026-01-30 14:38:23,917 - DEBUG - Requesting: http://10.0.0.131:8082/project/98447ac1-24d6-478e-adcc-dc791d752844 2026-01-30 14:38:23,951 - DEBUG - Requesting: http://10.0.0.131:8082/virtual-network/fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:23,959 - DEBUG - Requesting: http://10.0.0.131:8082/floating-ip-pool/7e71886d-38f4-4c01-9766-f702d7010323 2026-01-30 14:38:23,964 - DEBUG - Requesting: http://10.0.0.131:8082/domains 2026-01-30 14:38:23,967 - DEBUG - Requesting: http://10.0.0.131:8082/domain/bda8e106-cb10-4d85-94cf-36ec65254e20 2026-01-30 14:38:23,973 - DEBUG - Requesting: http://10.0.0.131:8082/project/98447ac1-24d6-478e-adcc-dc791d752844 2026-01-30 14:38:24,010 - DEBUG - Requesting: http://10.0.0.131:8082/virtual-network/fdb1c017-44e8-4fca-aaec-fdc0b0defbd7 2026-01-30 14:38:24,018 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-01-30 14:38:24,018 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-01-30 14:38:24,024 - DEBUG - Requesting: http://10.0.0.131:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-01-30 14:38:24,029 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-01-30 14:38:24,029 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-01-30 14:38:24,033 - DEBUG - Ingress ctest-nginx-ingress-97300417 not present 2026-01-30 14:38:24,034 - INFO - Creating Ingress ctest-nginx-ingress-97300417 2026-01-30 14:38:24,040 - INFO - Created Ingress ctest-nginx-ingress-97300417 2026-01-30 14:38:24,048 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-97300417 2026-01-30 14:38:27,053 - DEBUG - For Ingress ctest-nginx-ingress-97300417, Cluster IP: 10.47.255.249, External IPs ['155.66.214.60'] 2026-01-30 14:38:27,053 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-97300417 2026-01-30 14:38:27,054 - DEBUG - Requesting: http://10.0.0.131:8108/Snh_LoadbalancerDatabaseList?x=96a3b6ec-3503-4f7e-aa6f-48248e037bc1 2026-01-30 14:38:27,059 - INFO - Ingress ctest-nginx-ingress-97300417 with uuid 96a3b6ec-3503-4f7e-aa6f-48248e037bc1 found in kube manager 2026-01-30 14:38:27,060 - INFO - Ingress ctest-nginx-ingress-97300417 verification passed 2026-01-30 14:38:27,064 - DEBUG - Pod ctest-busybox-pod-06479372 not present 2026-01-30 14:38:27,065 - INFO - Creating Pod ctest-busybox-pod-06479372 2026-01-30 14:38:27,104 - DEBUG - Pod : ctest-busybox-pod-06479372 UUID is 76301924-d4f9-4a1c-867b-602aa25adec3 2026-01-30 14:38:27,115 - DEBUG - Pod ctest-nginx-pod-75763037 not in running state.Currently in Pending 2026-01-30 14:38:32,122 - INFO - Pod ctest-nginx-pod-75763037 is in running state.Got IP 10.47.255.251 2026-01-30 14:38:32,179 - DEBUG - Pod ctest-nginx-pod-75763037 has vmi 520451b0-fde9-11f0-8a09-fa163eae7302 2026-01-30 14:38:32,179 - INFO - Verified pod ctest-nginx-pod-75763037 in contrail-api 2026-01-30 14:38:32,179 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-01-30 14:38:32,202 - DEBUG - VMI 520451b0-fde9-11f0-8a09-fa163eae7302 is active in agent 10.0.0.136 2026-01-30 14:38:32,203 - INFO - Verified Pod ctest-nginx-pod-75763037 in agent 10.0.0.136 2026-01-30 14:38:32,203 - DEBUG - Requesting: http://10.0.0.131:8108/Snh_VirtualMachineDatabaseList?x=0367a583-8c19-4c28-80d1-a42d9e3ec986 2026-01-30 14:38:32,210 - INFO - Pod ctest-nginx-pod-75763037 with uuid 0367a583-8c19-4c28-80d1-a42d9e3ec986 found in kube manager 2026-01-30 14:38:32,210 - INFO - Pod ctest-nginx-pod-75763037 verification passed 2026-01-30 14:38:32,291 - DEBUG - [Pod ctest-nginx-pod-75763037] Cmd: echo ctest-nginx-pod-75763037 > /usr/share/nginx/html/index.html, Output: 2026-01-30 14:38:32,298 - INFO - Pod ctest-nginx-pod-59542138 is in running state.Got IP 10.47.255.250 2026-01-30 14:38:32,332 - DEBUG - Pod ctest-nginx-pod-59542138 has vmi 5256a10e-fde9-11f0-8a09-fa163eae7302 2026-01-30 14:38:32,333 - INFO - Verified pod ctest-nginx-pod-59542138 in contrail-api 2026-01-30 14:38:32,333 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-01-30 14:38:32,344 - DEBUG - VMI 5256a10e-fde9-11f0-8a09-fa163eae7302 is active in agent 10.0.0.136 2026-01-30 14:38:32,344 - INFO - Verified Pod ctest-nginx-pod-59542138 in agent 10.0.0.136 2026-01-30 14:38:32,345 - DEBUG - Requesting: http://10.0.0.131:8108/Snh_VirtualMachineDatabaseList?x=ba2705ef-233a-4d67-9f2b-8a9145b0a336 2026-01-30 14:38:32,349 - INFO - Pod ctest-nginx-pod-59542138 with uuid ba2705ef-233a-4d67-9f2b-8a9145b0a336 found in kube manager 2026-01-30 14:38:32,349 - INFO - Pod ctest-nginx-pod-59542138 verification passed 2026-01-30 14:38:32,410 - DEBUG - [Pod ctest-nginx-pod-59542138] Cmd: echo ctest-nginx-pod-59542138 > /usr/share/nginx/html/index.html, Output: 2026-01-30 14:38:32,416 - DEBUG - Pod ctest-busybox-pod-06479372 not in running state.Currently in Pending 2026-01-30 14:38:37,434 - INFO - Pod ctest-busybox-pod-06479372 is in running state.Got IP 10.47.255.248 2026-01-30 14:38:37,445 - DEBUG - Pod ctest-busybox-pod-06479372 has vmi 56b7b7f6-fde9-11f0-8a09-fa163eae7302 2026-01-30 14:38:37,445 - INFO - Verified pod ctest-busybox-pod-06479372 in contrail-api 2026-01-30 14:38:37,445 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-01-30 14:38:37,455 - DEBUG - VMI 56b7b7f6-fde9-11f0-8a09-fa163eae7302 is active in agent 10.0.0.136 2026-01-30 14:38:37,455 - INFO - Verified Pod ctest-busybox-pod-06479372 in agent 10.0.0.136 2026-01-30 14:38:37,455 - DEBUG - Requesting: http://10.0.0.131:8108/Snh_VirtualMachineDatabaseList?x=76301924-d4f9-4a1c-867b-602aa25adec3 2026-01-30 14:38:37,459 - INFO - Pod ctest-busybox-pod-06479372 with uuid 76301924-d4f9-4a1c-867b-602aa25adec3 found in kube manager 2026-01-30 14:38:37,460 - INFO - Pod ctest-busybox-pod-06479372 verification passed 2026-01-30 14:38:37,520 - DEBUG - [Pod ctest-busybox-pod-06479372] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-75763037 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-01-30 14:38:37,520 - DEBUG - [Pod Pod ctest-busybox-pod-06479372] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-01-30 14:38:37,520 - DEBUG - [Pod Pod ctest-busybox-pod-06479372] Cmd output: ctest-nginx-pod-75763037 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-01-30 14:38:37,583 - DEBUG - [Pod ctest-busybox-pod-06479372] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-59542138 2026-01-30 14:38:37,583 - DEBUG - [Pod Pod ctest-busybox-pod-06479372] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-01-30 14:38:37,583 - DEBUG - [Pod Pod ctest-busybox-pod-06479372] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-59542138 2026-01-30 14:38:37,583 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-75763037': 1, 'ctest-nginx-pod-59542138': 1} 2026-01-30 14:38:37,583 - WARNING - Nothing to delete parallely 2026-01-30 14:38:37,583 - INFO - Deleting pod default:ctest-busybox-pod-06479372 2026-01-30 14:38:37,596 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=76301924-d4f9-4a1c-867b-602aa25adec3 2026-01-30 14:38:37,603 - WARNING - Pod uuid 76301924-d4f9-4a1c-867b-602aa25adec3 is still seen in agent 10.0.0.136 VM list 2026-01-30 14:38:42,604 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=76301924-d4f9-4a1c-867b-602aa25adec3 2026-01-30 14:38:42,610 - WARNING - Pod uuid 76301924-d4f9-4a1c-867b-602aa25adec3 is still seen in agent 10.0.0.136 VM list 2026-01-30 14:38:47,611 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=76301924-d4f9-4a1c-867b-602aa25adec3 2026-01-30 14:38:47,619 - WARNING - Pod uuid 76301924-d4f9-4a1c-867b-602aa25adec3 is still seen in agent 10.0.0.136 VM list 2026-01-30 14:38:52,620 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=76301924-d4f9-4a1c-867b-602aa25adec3 2026-01-30 14:38:52,631 - WARNING - Pod uuid 76301924-d4f9-4a1c-867b-602aa25adec3 is still seen in agent 10.0.0.136 VM list 2026-01-30 14:38:57,631 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=76301924-d4f9-4a1c-867b-602aa25adec3 2026-01-30 14:38:57,641 - WARNING - Pod uuid 76301924-d4f9-4a1c-867b-602aa25adec3 is still seen in agent 10.0.0.136 VM list 2026-01-30 14:39:02,642 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=76301924-d4f9-4a1c-867b-602aa25adec3 2026-01-30 14:39:02,652 - WARNING - Pod uuid 76301924-d4f9-4a1c-867b-602aa25adec3 is still seen in agent 10.0.0.136 VM list 2026-01-30 14:39:07,652 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=76301924-d4f9-4a1c-867b-602aa25adec3 2026-01-30 14:39:07,660 - WARNING - Pod uuid 76301924-d4f9-4a1c-867b-602aa25adec3 is still seen in agent 10.0.0.136 VM list 2026-01-30 14:39:12,660 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=76301924-d4f9-4a1c-867b-602aa25adec3 2026-01-30 14:39:12,668 - DEBUG - Pod 76301924-d4f9-4a1c-867b-602aa25adec3 is not in agent 10.0.0.136 VM list 2026-01-30 14:39:12,668 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:56b7b7f6-fde9-11f0-8a09-fa163eae7302 2026-01-30 14:39:12,674 - DEBUG - VMI 56b7b7f6-fde9-11f0-8a09-fa163eae7302 is removed from agent 10.0.0.136 2026-01-30 14:39:12,674 - INFO - Verified that pod ctest-busybox-pod-06479372 is removed in agent 2026-01-30 14:39:12,674 - INFO - Deleting Ingress : ctest-nginx-ingress-97300417 2026-01-30 14:39:12,680 - DEBUG - Requesting: http://10.0.0.131:8108/Snh_LoadbalancerDatabaseList?x=96a3b6ec-3503-4f7e-aa6f-48248e037bc1 2026-01-30 14:39:12,685 - ERROR - Ingress ctest-nginx-ingress-97300417 with uuid 96a3b6ec-3503-4f7e-aa6f-48248e037bc1 still found in kube manager 2026-01-30 14:39:13,686 - DEBUG - Requesting: http://10.0.0.131:8108/Snh_LoadbalancerDatabaseList?x=96a3b6ec-3503-4f7e-aa6f-48248e037bc1 2026-01-30 14:39:13,691 - INFO - Ingress ctest-nginx-ingress-97300417 with uuid 96a3b6ec-3503-4f7e-aa6f-48248e037bc1 deleted successfully from kube manager 2026-01-30 14:39:13,691 - INFO - Verifications on Ingress %s cleanup passed 2026-01-30 14:39:13,691 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-01-30 14:39:13,833 - DEBUG - Requesting: http://10.0.0.131:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-01-30 14:39:13,841 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-01-30 14:39:13,841 - INFO - Deleting VN __public__ 2026-01-30 14:39:13,996 - DEBUG - Requesting: http://10.0.0.131:8082/routing-instance/bfed1a7d-5b6b-45b7-b0bd-461c9e8945f4 2026-01-30 14:39:14,005 - DEBUG - Response Code: 404 2026-01-30 14:39:14,005 - DEBUG - Requesting: http://10.0.0.131:8082/domains 2026-01-30 14:39:14,010 - DEBUG - Requesting: http://10.0.0.131:8082/domain/bda8e106-cb10-4d85-94cf-36ec65254e20 2026-01-30 14:39:14,017 - DEBUG - Requesting: http://10.0.0.131:8082/project/98447ac1-24d6-478e-adcc-dc791d752844 2026-01-30 14:39:14,079 - INFO - Validated that VN __public__ is not found in API Server 2026-01-30 14:39:14,079 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-01-30 14:39:14,090 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 14:39:14,100 - DEBUG - VN __public__ is not present in Agent 10.0.0.136 2026-01-30 14:39:14,100 - INFO - Validated that VN __public__ is not in any agent 2026-01-30 14:39:14,100 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 14:39:14,108 - DEBUG - Requesting: http://10.0.0.131:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-01-30 14:39:14,113 - DEBUG - Requesting: http://10.0.0.131:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-01-30 14:39:14,117 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-01-30 14:39:14,117 - INFO - Deleting pod default:ctest-nginx-pod-59542138 2026-01-30 14:39:14,139 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=ba2705ef-233a-4d67-9f2b-8a9145b0a336 2026-01-30 14:39:14,158 - WARNING - Pod uuid ba2705ef-233a-4d67-9f2b-8a9145b0a336 is still seen in agent 10.0.0.136 VM list 2026-01-30 14:39:19,159 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=ba2705ef-233a-4d67-9f2b-8a9145b0a336 2026-01-30 14:39:19,173 - DEBUG - Pod ba2705ef-233a-4d67-9f2b-8a9145b0a336 is not in agent 10.0.0.136 VM list 2026-01-30 14:39:19,173 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5256a10e-fde9-11f0-8a09-fa163eae7302 2026-01-30 14:39:19,180 - DEBUG - VMI 5256a10e-fde9-11f0-8a09-fa163eae7302 is removed from agent 10.0.0.136 2026-01-30 14:39:19,180 - INFO - Verified that pod ctest-nginx-pod-59542138 is removed in agent 2026-01-30 14:39:19,181 - INFO - Deleting pod default:ctest-nginx-pod-75763037 2026-01-30 14:39:19,198 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=0367a583-8c19-4c28-80d1-a42d9e3ec986 2026-01-30 14:39:19,212 - WARNING - Pod uuid 0367a583-8c19-4c28-80d1-a42d9e3ec986 is still seen in agent 10.0.0.136 VM list 2026-01-30 14:39:24,214 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=0367a583-8c19-4c28-80d1-a42d9e3ec986 2026-01-30 14:39:24,220 - DEBUG - Pod 0367a583-8c19-4c28-80d1-a42d9e3ec986 is not in agent 10.0.0.136 VM list 2026-01-30 14:39:24,220 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:520451b0-fde9-11f0-8a09-fa163eae7302 2026-01-30 14:39:24,226 - DEBUG - VMI 520451b0-fde9-11f0-8a09-fa163eae7302 is removed from agent 10.0.0.136 2026-01-30 14:39:24,226 - INFO - Verified that pod ctest-nginx-pod-75763037 is removed in agent 2026-01-30 14:39:24,227 - INFO - Deleting service : ctest-nginx-svc-37840724 2026-01-30 14:39:24,475 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_AgentXmppConnectionStatusReq? 2026-01-30 14:39:24,482 - DEBUG - No XMPP flaps were noticed during the test