2025-10-21 20:46:20,847 - DEBUG - [10.0.0.242]: Running cmd : hostname 2025-10-21 20:46:21,098 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1585-1 2025-10-21 20:46:21,098 - DEBUG - [10.0.0.242]: Running cmd : hostname -f 2025-10-21 20:46:21,132 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1585-1 2025-10-21 20:46:21,132 - DEBUG - [10.0.0.242]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-21 20:46:21,255 - DEBUG - Output : NAMES contrail_test_xZNRRtWam contrail-kubernetes-master-kubemanager-1 analytics_snmp-topology-1 analytics_snmp-snmp-collector-1 analytics_snmp-nodemgr-1 analytics_alarm-kafka-1 analytics_alarm-nodemgr-1 analytics_alarm-alarm-gen-1 analytics-collector-1 analytics-api-1 analytics-nodemgr-1 analytics_database-query-engine-1 analytics_database-nodemgr-1 analytics_database-cassandra-1 redis-redis-1 webui-web-1 webui-job-1 control-named-1 control-control-1 control-dns-1 control-nodemgr-1 config_database-nodemgr-1 config_database-rabbitmq-1 config_database-zookeeper-1 config_database-cassandra-1 config_api-devicemgr-1 config_api-api-1 config_api-nodemgr-1 config_api-svcmonitor-1 config_api-dnsmasq-1 config_api-schema-1 2025-10-21 20:46:21,256 - DEBUG - [10.0.0.242]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-10-21 20:46:21,302 - DEBUG - Output : 127.0.0.1/8 10.0.0.242/24 252.242.0.1/8 10.249.105.1/24 172.17.0.1/16 2025-10-21 20:46:21,303 - DEBUG - [10.0.0.242]: Running cmd : getent hosts 10.0.0.242 | head -n 1 | awk '{print $2}' 2025-10-21 20:46:21,349 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1585-1 2025-10-21 20:46:21,440 - DEBUG - [10.0.0.33]: Running cmd : hostname 2025-10-21 20:46:21,674 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1585-1 2025-10-21 20:46:21,674 - DEBUG - [10.0.0.33]: Running cmd : hostname -f 2025-10-21 20:46:21,743 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1585-1 2025-10-21 20:46:21,744 - DEBUG - [10.0.0.33]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-21 20:46:21,799 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2025-10-21 20:46:21,799 - DEBUG - [10.0.0.33]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-10-21 20:46:21,867 - DEBUG - Output : 10.0.0.33/24 2025-10-21 20:46:21,868 - DEBUG - [10.0.0.33]: Running cmd : getent hosts 10.0.0.33 | head -n 1 | awk '{print $2}' 2025-10-21 20:46:21,912 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1585-1 2025-10-21 20:46:22,128 - INFO - Using existing project ['default-domain', 'k8s-default'](90f74e27-1327-438b-ba49-3ce83523a102) 2025-10-21 20:46:22,264 - INFO - Using existing project ['default-domain', 'k8s-default'](90f74e27-1327-438b-ba49-3ce83523a102) 2025-10-21 20:46:22,403 - INFO - ================================================================================ 2025-10-21 20:46:22,410 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-10-21 20:46:22,410 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-10-21 20:46:22,645 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-21 20:46:22,652 - DEBUG - Nothing to compare xmpp stats {'10.0.0.33': {'10.0.0.242': '0'}} with 2025-10-21 20:46:22,652 - INFO - Initial checks done. Running the testcase now 2025-10-21 20:46:22,652 - INFO - 2025-10-21 20:46:22,657 - INFO - Namespace default already exists 2025-10-21 20:46:22,657 - DEBUG - [10.0.0.242]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-10-21 20:46:22,801 - DEBUG - Output : cluster_project={} 2025-10-21 20:46:22,892 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-10-21 20:46:22,892 - DEBUG - Project uuid in Contrail is 90f74e27-1327-438b-ba49-3ce83523a102 2025-10-21 20:46:22,892 - DEBUG - [10.0.0.242]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-10-21 20:46:23,020 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-10-21 20:46:23,020 - DEBUG - Requesting: http://10.0.0.242:8108/Snh_NamespaceDatabaseList?namespace_uuid=07760def-14d8-4047-990e-e8becac564a9 2025-10-21 20:46:23,024 - INFO - Namespace default with uuid 07760def-14d8-4047-990e-e8becac564a9 found in kube manager 2025-10-21 20:46:23,025 - INFO - Namespace default verification passed 2025-10-21 20:46:23,027 - DEBUG - Service ctest-nginx-svc-69120127 not present 2025-10-21 20:46:23,028 - INFO - Creating service ctest-nginx-svc-69120127 2025-10-21 20:46:23,052 - INFO - Created Service ctest-nginx-svc-69120127 2025-10-21 20:46:23,083 - DEBUG - Pod ctest-nginx-pod-39210086 not present 2025-10-21 20:46:23,084 - INFO - Creating Pod ctest-nginx-pod-39210086 2025-10-21 20:46:23,143 - DEBUG - Pod : ctest-nginx-pod-39210086 UUID is 3972f501-038c-4694-910c-65960e92a781 2025-10-21 20:46:23,148 - DEBUG - Pod ctest-nginx-pod-51041430 not present 2025-10-21 20:46:23,149 - INFO - Creating Pod ctest-nginx-pod-51041430 2025-10-21 20:46:23,171 - DEBUG - Pod : ctest-nginx-pod-51041430 UUID is 0f8dde1a-548e-4ff2-89ed-84672a640afd 2025-10-21 20:46:23,555 - INFO - Created VN __public__, UUID :43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:23,821 - DEBUG - Requesting: http://10.0.0.242:8082/domains 2025-10-21 20:46:23,831 - DEBUG - Requesting: http://10.0.0.242:8082/domain/9214d1a4-8b92-4307-8f57-47df58f66df9 2025-10-21 20:46:23,877 - DEBUG - Requesting: http://10.0.0.242:8082/project/90f74e27-1327-438b-ba49-3ce83523a102 2025-10-21 20:46:23,979 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-network/43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:24,010 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-network/43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:24,027 - DEBUG - Requesting: http://10.0.0.242:8082/routing-instance/58175872-8d44-4384-a869-9be4046100d1 2025-10-21 20:46:24,041 - DEBUG - Requesting: http://10.0.0.242:8082/routing-instance/58175872-8d44-4384-a869-9be4046100d1 2025-10-21 20:46:24,077 - DEBUG - Route Targets: [] 2025-10-21 20:46:24,077 - DEBUG - RT names not yet present for VN __public__ 2025-10-21 20:46:29,078 - DEBUG - Requesting: http://10.0.0.242:8082/domains 2025-10-21 20:46:29,083 - DEBUG - Requesting: http://10.0.0.242:8082/domain/9214d1a4-8b92-4307-8f57-47df58f66df9 2025-10-21 20:46:29,089 - DEBUG - Requesting: http://10.0.0.242:8082/project/90f74e27-1327-438b-ba49-3ce83523a102 2025-10-21 20:46:29,129 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-network/43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:29,136 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-network/43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:29,144 - DEBUG - Requesting: http://10.0.0.242:8082/routing-instance/58175872-8d44-4384-a869-9be4046100d1 2025-10-21 20:46:29,150 - DEBUG - Requesting: http://10.0.0.242:8082/routing-instance/58175872-8d44-4384-a869-9be4046100d1 2025-10-21 20:46:29,156 - DEBUG - Requesting: http://10.0.0.242:8082/route-target/86103004-fd96-4575-975c-d3bfa37a35eb 2025-10-21 20:46:29,160 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-21 20:46:29,161 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-network/43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:29,167 - DEBUG - Requesting: http://10.0.0.242:8082/routing-instance/58175872-8d44-4384-a869-9be4046100d1 2025-10-21 20:46:29,178 - INFO - Verified VN network id 13 for VN 43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:29,178 - INFO - Verifications in API Server for VN __public__ passed 2025-10-21 20:46:29,178 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-network/43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:29,185 - DEBUG - Requesting: http://10.0.0.242:8082/routing-instance/58175872-8d44-4384-a869-9be4046100d1 2025-10-21 20:46:29,190 - DEBUG - Requesting: http://10.0.0.242:8082/routing-instance/58175872-8d44-4384-a869-9be4046100d1 2025-10-21 20:46:29,195 - DEBUG - Requesting: http://10.0.0.242:8082/route-target/86103004-fd96-4575-975c-d3bfa37a35eb 2025-10-21 20:46:29,200 - DEBUG - Requesting: http://10.0.0.242:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-21 20:46:29,206 - DEBUG - Control-node 10.0.0.242 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4851571288346740979', 'uuid-lslong': '12806707017946752775'}, 'enable': 'true', 'created': '2025-10-21T20:46:23', 'last-modified': '2025-10-21T20:46:23', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.595969'} 2025-10-21 20:46:29,206 - DEBUG - Requesting: http://10.0.0.242:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-21 20:46:29,216 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-21 20:46:29,216 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-10-21 20:46:29,217 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-10-21 20:46:29,217 - DEBUG - Requesting: http://10.0.0.242:8082/domains 2025-10-21 20:46:29,221 - DEBUG - Requesting: http://10.0.0.242:8082/domain/9214d1a4-8b92-4307-8f57-47df58f66df9 2025-10-21 20:46:29,228 - DEBUG - Requesting: http://10.0.0.242:8082/project/90f74e27-1327-438b-ba49-3ce83523a102 2025-10-21 20:46:29,278 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-network/43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:29,286 - DEBUG - =>VN __public__ has no policy to be verified 2025-10-21 20:46:29,286 - DEBUG - Verifying the vn in opserver 2025-10-21 20:46:29,286 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.242 2025-10-21 20:46:29,286 - DEBUG - Requesting: http://10.0.0.242:8081/analytics/uves/virtual-networks 2025-10-21 20:46:29,290 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.242:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-10-21 20:46:29,290 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-10-21 20:46:29,290 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-21 20:46:29,298 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-21 20:46:29,303 - DEBUG - Do not have enough data to verify VN in agent 2025-10-21 20:46:29,303 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-21 20:46:29,308 - DEBUG - VRF ids for VN __public__: {} 2025-10-21 20:46:29,412 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-10-21 20:46:29,515 - DEBUG - Requesting: http://10.0.0.242:8082/domains 2025-10-21 20:46:29,533 - DEBUG - Requesting: http://10.0.0.242:8082/domain/9214d1a4-8b92-4307-8f57-47df58f66df9 2025-10-21 20:46:29,541 - DEBUG - Requesting: http://10.0.0.242:8082/project/90f74e27-1327-438b-ba49-3ce83523a102 2025-10-21 20:46:29,586 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-network/43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:29,597 - DEBUG - Requesting: http://10.0.0.242:8082/floating-ip-pool/1871a76b-1379-4c71-a635-6781c6020ff4 2025-10-21 20:46:29,602 - DEBUG - Requesting: http://10.0.0.242:8082/domains 2025-10-21 20:46:29,608 - DEBUG - Requesting: http://10.0.0.242:8082/domain/9214d1a4-8b92-4307-8f57-47df58f66df9 2025-10-21 20:46:29,613 - DEBUG - Requesting: http://10.0.0.242:8082/project/90f74e27-1327-438b-ba49-3ce83523a102 2025-10-21 20:46:29,648 - DEBUG - Requesting: http://10.0.0.242:8082/virtual-network/43543e61-cc64-48f3-b1ba-902e1337e707 2025-10-21 20:46:29,660 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-10-21 20:46:29,660 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-10-21 20:46:29,673 - DEBUG - Requesting: http://10.0.0.242:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-21 20:46:29,680 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-10-21 20:46:29,680 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-10-21 20:46:29,683 - DEBUG - Ingress ctest-nginx-ingress-34190726 not present 2025-10-21 20:46:29,684 - INFO - Creating Ingress ctest-nginx-ingress-34190726 2025-10-21 20:46:29,693 - INFO - Created Ingress ctest-nginx-ingress-34190726 2025-10-21 20:46:29,697 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-34190726 2025-10-21 20:46:32,702 - DEBUG - For Ingress ctest-nginx-ingress-34190726, Cluster IP: 10.47.255.249, External IPs ['116.203.33.188'] 2025-10-21 20:46:32,703 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-34190726 2025-10-21 20:46:32,703 - DEBUG - Requesting: http://10.0.0.242:8108/Snh_LoadbalancerDatabaseList?x=ccd658bf-ba5c-477f-9fee-0613e6c75d16 2025-10-21 20:46:32,707 - INFO - Ingress ctest-nginx-ingress-34190726 with uuid ccd658bf-ba5c-477f-9fee-0613e6c75d16 found in kube manager 2025-10-21 20:46:32,708 - INFO - Ingress ctest-nginx-ingress-34190726 verification passed 2025-10-21 20:46:32,723 - DEBUG - Pod ctest-busybox-pod-27619906 not present 2025-10-21 20:46:32,724 - INFO - Creating Pod ctest-busybox-pod-27619906 2025-10-21 20:46:32,759 - DEBUG - Pod : ctest-busybox-pod-27619906 UUID is b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:46:32,767 - DEBUG - Pod ctest-nginx-pod-39210086 not in running state.Currently in Pending 2025-10-21 20:46:37,774 - INFO - Pod ctest-nginx-pod-39210086 is in running state.Got IP 10.47.255.251 2025-10-21 20:46:37,828 - DEBUG - Pod ctest-nginx-pod-39210086 has vmi 025f118c-aebf-11f0-839a-fa163e6fefa3 2025-10-21 20:46:37,828 - INFO - Verified pod ctest-nginx-pod-39210086 in contrail-api 2025-10-21 20:46:37,828 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-21 20:46:37,839 - DEBUG - VMI 025f118c-aebf-11f0-839a-fa163e6fefa3 is active in agent 10.0.0.33 2025-10-21 20:46:37,839 - INFO - Verified Pod ctest-nginx-pod-39210086 in agent 10.0.0.33 2025-10-21 20:46:37,839 - DEBUG - Requesting: http://10.0.0.242:8108/Snh_VirtualMachineDatabaseList?x=3972f501-038c-4694-910c-65960e92a781 2025-10-21 20:46:37,843 - INFO - Pod ctest-nginx-pod-39210086 with uuid 3972f501-038c-4694-910c-65960e92a781 found in kube manager 2025-10-21 20:46:37,843 - INFO - Pod ctest-nginx-pod-39210086 verification passed 2025-10-21 20:46:37,916 - DEBUG - [Pod ctest-nginx-pod-39210086] Cmd: echo ctest-nginx-pod-39210086 > /usr/share/nginx/html/index.html, Output: 2025-10-21 20:46:37,925 - INFO - Pod ctest-nginx-pod-51041430 is in running state.Got IP 10.47.255.250 2025-10-21 20:46:37,940 - DEBUG - Pod ctest-nginx-pod-51041430 has vmi 02cbc39a-aebf-11f0-839a-fa163e6fefa3 2025-10-21 20:46:37,940 - INFO - Verified pod ctest-nginx-pod-51041430 in contrail-api 2025-10-21 20:46:37,940 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-21 20:46:37,951 - DEBUG - VMI 02cbc39a-aebf-11f0-839a-fa163e6fefa3 is active in agent 10.0.0.33 2025-10-21 20:46:37,951 - INFO - Verified Pod ctest-nginx-pod-51041430 in agent 10.0.0.33 2025-10-21 20:46:37,951 - DEBUG - Requesting: http://10.0.0.242:8108/Snh_VirtualMachineDatabaseList?x=0f8dde1a-548e-4ff2-89ed-84672a640afd 2025-10-21 20:46:37,956 - INFO - Pod ctest-nginx-pod-51041430 with uuid 0f8dde1a-548e-4ff2-89ed-84672a640afd found in kube manager 2025-10-21 20:46:37,956 - INFO - Pod ctest-nginx-pod-51041430 verification passed 2025-10-21 20:46:38,014 - DEBUG - [Pod ctest-nginx-pod-51041430] Cmd: echo ctest-nginx-pod-51041430 > /usr/share/nginx/html/index.html, Output: 2025-10-21 20:46:38,021 - DEBUG - Pod ctest-busybox-pod-27619906 not in running state.Currently in Pending 2025-10-21 20:46:43,028 - DEBUG - Pod ctest-busybox-pod-27619906 not in running state.Currently in Pending 2025-10-21 20:46:48,035 - INFO - Pod ctest-busybox-pod-27619906 is in running state.Got IP 10.47.255.248 2025-10-21 20:46:48,046 - DEBUG - Pod ctest-busybox-pod-27619906 has vmi 071bbe00-aebf-11f0-839a-fa163e6fefa3 2025-10-21 20:46:48,047 - INFO - Verified pod ctest-busybox-pod-27619906 in contrail-api 2025-10-21 20:46:48,047 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-21 20:46:48,056 - DEBUG - VMI 071bbe00-aebf-11f0-839a-fa163e6fefa3 is active in agent 10.0.0.33 2025-10-21 20:46:48,057 - INFO - Verified Pod ctest-busybox-pod-27619906 in agent 10.0.0.33 2025-10-21 20:46:48,057 - DEBUG - Requesting: http://10.0.0.242:8108/Snh_VirtualMachineDatabaseList?x=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:46:48,060 - INFO - Pod ctest-busybox-pod-27619906 with uuid b4c08680-84ed-4b7d-8f49-56e53728fe70 found in kube manager 2025-10-21 20:46:48,060 - INFO - Pod ctest-busybox-pod-27619906 verification passed 2025-10-21 20:46:48,108 - DEBUG - [Pod ctest-busybox-pod-27619906] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) ctest-nginx-pod-39210086 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-21 20:46:48,108 - DEBUG - [Pod Pod ctest-busybox-pod-27619906] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2025-10-21 20:46:48,108 - DEBUG - [Pod Pod ctest-busybox-pod-27619906] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) ctest-nginx-pod-39210086 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-21 20:46:48,165 - DEBUG - [Pod ctest-busybox-pod-27619906] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-39210086 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-21 20:46:48,165 - DEBUG - [Pod Pod ctest-busybox-pod-27619906] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2025-10-21 20:46:48,165 - DEBUG - [Pod Pod ctest-busybox-pod-27619906] Cmd output: ctest-nginx-pod-39210086 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-21 20:46:48,201 - DEBUG - [Pod ctest-busybox-pod-27619906] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-51041430 2025-10-21 20:46:48,201 - DEBUG - [Pod Pod ctest-busybox-pod-27619906] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2025-10-21 20:46:48,201 - DEBUG - [Pod Pod ctest-busybox-pod-27619906] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-51041430 2025-10-21 20:46:48,201 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-39210086': 2, 'ctest-nginx-pod-51041430': 1} 2025-10-21 20:46:48,201 - WARNING - Nothing to delete parallely 2025-10-21 20:46:48,202 - INFO - Deleting pod default:ctest-busybox-pod-27619906 2025-10-21 20:46:48,212 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:46:48,218 - WARNING - Pod uuid b4c08680-84ed-4b7d-8f49-56e53728fe70 is still seen in agent 10.0.0.33 VM list 2025-10-21 20:46:53,219 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:46:53,225 - WARNING - Pod uuid b4c08680-84ed-4b7d-8f49-56e53728fe70 is still seen in agent 10.0.0.33 VM list 2025-10-21 20:46:58,226 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:46:58,232 - WARNING - Pod uuid b4c08680-84ed-4b7d-8f49-56e53728fe70 is still seen in agent 10.0.0.33 VM list 2025-10-21 20:47:03,233 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:47:03,240 - WARNING - Pod uuid b4c08680-84ed-4b7d-8f49-56e53728fe70 is still seen in agent 10.0.0.33 VM list 2025-10-21 20:47:08,241 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:47:08,248 - WARNING - Pod uuid b4c08680-84ed-4b7d-8f49-56e53728fe70 is still seen in agent 10.0.0.33 VM list 2025-10-21 20:47:13,249 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:47:13,255 - WARNING - Pod uuid b4c08680-84ed-4b7d-8f49-56e53728fe70 is still seen in agent 10.0.0.33 VM list 2025-10-21 20:47:18,256 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:47:18,263 - WARNING - Pod uuid b4c08680-84ed-4b7d-8f49-56e53728fe70 is still seen in agent 10.0.0.33 VM list 2025-10-21 20:47:23,264 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:47:23,272 - WARNING - Pod uuid b4c08680-84ed-4b7d-8f49-56e53728fe70 is still seen in agent 10.0.0.33 VM list 2025-10-21 20:47:28,273 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:47:28,279 - WARNING - Pod uuid b4c08680-84ed-4b7d-8f49-56e53728fe70 is still seen in agent 10.0.0.33 VM list 2025-10-21 20:47:33,279 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=b4c08680-84ed-4b7d-8f49-56e53728fe70 2025-10-21 20:47:33,285 - DEBUG - Pod b4c08680-84ed-4b7d-8f49-56e53728fe70 is not in agent 10.0.0.33 VM list 2025-10-21 20:47:33,285 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:071bbe00-aebf-11f0-839a-fa163e6fefa3 2025-10-21 20:47:33,290 - DEBUG - VMI 071bbe00-aebf-11f0-839a-fa163e6fefa3 is removed from agent 10.0.0.33 2025-10-21 20:47:33,290 - INFO - Verified that pod ctest-busybox-pod-27619906 is removed in agent 2025-10-21 20:47:33,290 - INFO - Deleting Ingress : ctest-nginx-ingress-34190726 2025-10-21 20:47:33,307 - DEBUG - Requesting: http://10.0.0.242:8108/Snh_LoadbalancerDatabaseList?x=ccd658bf-ba5c-477f-9fee-0613e6c75d16 2025-10-21 20:47:33,315 - ERROR - Ingress ctest-nginx-ingress-34190726 with uuid ccd658bf-ba5c-477f-9fee-0613e6c75d16 still found in kube manager 2025-10-21 20:47:34,316 - DEBUG - Requesting: http://10.0.0.242:8108/Snh_LoadbalancerDatabaseList?x=ccd658bf-ba5c-477f-9fee-0613e6c75d16 2025-10-21 20:47:34,320 - INFO - Ingress ctest-nginx-ingress-34190726 with uuid ccd658bf-ba5c-477f-9fee-0613e6c75d16 deleted successfully from kube manager 2025-10-21 20:47:34,320 - INFO - Verifications on Ingress %s cleanup passed 2025-10-21 20:47:34,320 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-10-21 20:47:34,424 - DEBUG - Requesting: http://10.0.0.242:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-21 20:47:34,429 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-10-21 20:47:34,429 - INFO - Deleting VN __public__ 2025-10-21 20:47:34,550 - DEBUG - Requesting: http://10.0.0.242:8082/routing-instance/58175872-8d44-4384-a869-9be4046100d1 2025-10-21 20:47:34,556 - DEBUG - Response Code: 404 2025-10-21 20:47:34,557 - DEBUG - Requesting: http://10.0.0.242:8082/domains 2025-10-21 20:47:34,561 - DEBUG - Requesting: http://10.0.0.242:8082/domain/9214d1a4-8b92-4307-8f57-47df58f66df9 2025-10-21 20:47:34,573 - DEBUG - Requesting: http://10.0.0.242:8082/project/90f74e27-1327-438b-ba49-3ce83523a102 2025-10-21 20:47:34,644 - INFO - Validated that VN __public__ is not found in API Server 2025-10-21 20:47:34,645 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-10-21 20:47:34,651 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-21 20:47:34,660 - DEBUG - VN __public__ is not present in Agent 10.0.0.33 2025-10-21 20:47:34,660 - INFO - Validated that VN __public__ is not in any agent 2025-10-21 20:47:34,661 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-21 20:47:34,675 - DEBUG - Requesting: http://10.0.0.242:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-21 20:47:34,685 - DEBUG - Requesting: http://10.0.0.242:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-21 20:47:34,694 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-10-21 20:47:34,694 - INFO - Deleting pod default:ctest-nginx-pod-51041430 2025-10-21 20:47:34,722 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=0f8dde1a-548e-4ff2-89ed-84672a640afd 2025-10-21 20:47:34,751 - WARNING - Pod uuid 0f8dde1a-548e-4ff2-89ed-84672a640afd is still seen in agent 10.0.0.33 VM list 2025-10-21 20:47:39,752 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=0f8dde1a-548e-4ff2-89ed-84672a640afd 2025-10-21 20:47:39,758 - DEBUG - Pod 0f8dde1a-548e-4ff2-89ed-84672a640afd is not in agent 10.0.0.33 VM list 2025-10-21 20:47:39,758 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:02cbc39a-aebf-11f0-839a-fa163e6fefa3 2025-10-21 20:47:39,764 - DEBUG - VMI 02cbc39a-aebf-11f0-839a-fa163e6fefa3 is removed from agent 10.0.0.33 2025-10-21 20:47:39,764 - INFO - Verified that pod ctest-nginx-pod-51041430 is removed in agent 2025-10-21 20:47:39,764 - INFO - Deleting pod default:ctest-nginx-pod-39210086 2025-10-21 20:47:39,776 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3972f501-038c-4694-910c-65960e92a781 2025-10-21 20:47:39,784 - WARNING - Pod uuid 3972f501-038c-4694-910c-65960e92a781 is still seen in agent 10.0.0.33 VM list 2025-10-21 20:47:44,786 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3972f501-038c-4694-910c-65960e92a781 2025-10-21 20:47:44,792 - DEBUG - Pod 3972f501-038c-4694-910c-65960e92a781 is not in agent 10.0.0.33 VM list 2025-10-21 20:47:44,792 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:025f118c-aebf-11f0-839a-fa163e6fefa3 2025-10-21 20:47:44,797 - DEBUG - VMI 025f118c-aebf-11f0-839a-fa163e6fefa3 is removed from agent 10.0.0.33 2025-10-21 20:47:44,797 - INFO - Verified that pod ctest-nginx-pod-39210086 is removed in agent 2025-10-21 20:47:44,798 - INFO - Deleting service : ctest-nginx-svc-69120127 2025-10-21 20:47:45,002 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-21 20:47:45,009 - DEBUG - No XMPP flaps were noticed during the test