2026-02-09 20:22:41,747 - DEBUG - [10.0.0.33]: Running cmd : hostname 2026-02-09 20:22:41,987 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1865-1 2026-02-09 20:22:41,987 - DEBUG - [10.0.0.33]: Running cmd : hostname -f 2026-02-09 20:22:42,021 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1865-1 2026-02-09 20:22:42,021 - DEBUG - [10.0.0.33]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-09 20:22:42,122 - DEBUG - Output : NAMES contrail_test_KIaZu7rZb contrail-kubernetes-master-kubemanager-1 webui-web-1 webui-job-1 control-named-1 control-nodemgr-1 control-dns-1 control-control-1 config_database-cassandra-1 config_database-zookeeper-1 config_database-nodemgr-1 config_database-rabbitmq-1 config_api-devicemgr-1 config_api-api-1 config_api-svcmonitor-1 config_api-nodemgr-1 config_api-schema-1 config_api-dnsmasq-1 redis-redis-1 analytics_snmp-snmp-collector-1 analytics_snmp-nodemgr-1 analytics_snmp-topology-1 analytics_alarm-kafka-1 analytics_alarm-alarm-gen-1 analytics_alarm-nodemgr-1 analytics-nodemgr-1 analytics-api-1 analytics-collector-1 analytics_database-query-engine-1 analytics_database-nodemgr-1 analytics_database-cassandra-1 2026-02-09 20:22:42,123 - DEBUG - [10.0.0.33]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-02-09 20:22:42,156 - DEBUG - Output : 127.0.0.1/8 10.0.0.33/24 252.33.0.1/8 10.209.61.1/24 172.17.0.1/16 2026-02-09 20:22:42,157 - DEBUG - [10.0.0.33]: Running cmd : getent hosts 10.0.0.33 | head -n 1 | awk '{print $2}' 2026-02-09 20:22:42,241 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1865-1 2026-02-09 20:22:42,288 - DEBUG - [10.0.0.19]: Running cmd : hostname 2026-02-09 20:22:42,687 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1865-1 2026-02-09 20:22:42,687 - DEBUG - [10.0.0.19]: Running cmd : hostname -f 2026-02-09 20:22:42,763 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1865-1 2026-02-09 20:22:42,763 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-09 20:22:42,932 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2026-02-09 20:22:42,933 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-02-09 20:22:42,999 - DEBUG - Output : 10.0.0.19/24 2026-02-09 20:22:42,999 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.0.0.19 | head -n 1 | awk '{print $2}' 2026-02-09 20:22:43,066 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1865-1 2026-02-09 20:22:43,129 - INFO - Using existing project ['default-domain', 'k8s-default'](bfa9c0dc-1c9d-4d40-8bb4-c298630aa513) 2026-02-09 20:22:43,254 - INFO - Using existing project ['default-domain', 'k8s-default'](bfa9c0dc-1c9d-4d40-8bb4-c298630aa513) 2026-02-09 20:22:43,359 - INFO - ================================================================================ 2026-02-09 20:22:43,359 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-02-09 20:22:43,360 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-02-09 20:22:43,622 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-09 20:22:43,628 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.0.0.33': '0'}} with 2026-02-09 20:22:43,629 - INFO - Initial checks done. Running the testcase now 2026-02-09 20:22:43,629 - INFO - 2026-02-09 20:22:43,633 - INFO - Namespace default already exists 2026-02-09 20:22:43,633 - DEBUG - [10.0.0.33]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-09 20:22:43,749 - DEBUG - Output : cluster_project={} 2026-02-09 20:22:43,791 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-02-09 20:22:43,791 - DEBUG - Project uuid in Contrail is bfa9c0dc-1c9d-4d40-8bb4-c298630aa513 2026-02-09 20:22:43,791 - DEBUG - [10.0.0.33]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-02-09 20:22:43,957 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-02-09 20:22:43,958 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_NamespaceDatabaseList?namespace_uuid=77a3e8a9-29f6-45d6-ae8c-e001c4f1b3ec 2026-02-09 20:22:43,963 - INFO - Namespace default with uuid 77a3e8a9-29f6-45d6-ae8c-e001c4f1b3ec found in kube manager 2026-02-09 20:22:43,963 - INFO - Namespace default verification passed 2026-02-09 20:22:43,966 - DEBUG - Service ctest-nginx-svc-95681849 not present 2026-02-09 20:22:43,967 - INFO - Creating service ctest-nginx-svc-95681849 2026-02-09 20:22:43,996 - INFO - Created Service ctest-nginx-svc-95681849 2026-02-09 20:22:44,001 - DEBUG - Pod ctest-nginx-pod-43319799 not present 2026-02-09 20:22:44,002 - INFO - Creating Pod ctest-nginx-pod-43319799 2026-02-09 20:22:44,070 - DEBUG - Pod : ctest-nginx-pod-43319799 UUID is afe8ff22-b452-4c21-b6d0-8f2f945969ed 2026-02-09 20:22:44,075 - DEBUG - Pod ctest-nginx-pod-64627090 not present 2026-02-09 20:22:44,076 - INFO - Creating Pod ctest-nginx-pod-64627090 2026-02-09 20:22:44,110 - DEBUG - Pod : ctest-nginx-pod-64627090 UUID is 2806cf5c-cfe0-40d4-aff1-4dd74e87be9c 2026-02-09 20:22:44,289 - INFO - Created VN __public__, UUID :562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:44,609 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-09 20:22:44,615 - DEBUG - Requesting: http://10.0.0.33:8082/domain/31d0b242-908a-4dc4-83fa-cb50f11c1ca4 2026-02-09 20:22:44,632 - DEBUG - Requesting: http://10.0.0.33:8082/project/bfa9c0dc-1c9d-4d40-8bb4-c298630aa513 2026-02-09 20:22:44,768 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:44,792 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:44,832 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/5f1719d3-ee5a-4241-83bb-a686d04e9308 2026-02-09 20:22:44,845 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/5f1719d3-ee5a-4241-83bb-a686d04e9308 2026-02-09 20:22:44,867 - DEBUG - Route Targets: [] 2026-02-09 20:22:44,867 - DEBUG - RT names not yet present for VN __public__ 2026-02-09 20:22:49,869 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-09 20:22:49,874 - DEBUG - Requesting: http://10.0.0.33:8082/domain/31d0b242-908a-4dc4-83fa-cb50f11c1ca4 2026-02-09 20:22:49,883 - DEBUG - Requesting: http://10.0.0.33:8082/project/bfa9c0dc-1c9d-4d40-8bb4-c298630aa513 2026-02-09 20:22:49,927 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:49,937 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:49,945 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/5f1719d3-ee5a-4241-83bb-a686d04e9308 2026-02-09 20:22:49,953 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/5f1719d3-ee5a-4241-83bb-a686d04e9308 2026-02-09 20:22:49,962 - DEBUG - Requesting: http://10.0.0.33:8082/route-target/2ac8b3b1-2419-4f67-b803-59adf16b3ed6 2026-02-09 20:22:49,968 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-09 20:22:49,968 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:49,978 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/5f1719d3-ee5a-4241-83bb-a686d04e9308 2026-02-09 20:22:50,004 - INFO - Verified VN network id 13 for VN 562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:50,004 - INFO - Verifications in API Server for VN __public__ passed 2026-02-09 20:22:50,004 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:50,016 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/5f1719d3-ee5a-4241-83bb-a686d04e9308 2026-02-09 20:22:50,023 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/5f1719d3-ee5a-4241-83bb-a686d04e9308 2026-02-09 20:22:50,029 - DEBUG - Requesting: http://10.0.0.33:8082/route-target/2ac8b3b1-2419-4f67-b803-59adf16b3ed6 2026-02-09 20:22:50,035 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-09 20:22:50,043 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6209128287436686652', 'uuid-lslong': '9498902442462199969'}, 'enable': 'true', 'created': '2026-02-09T20:22:44', 'last-modified': '2026-02-09T20:22:44', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.536109'} 2026-02-09 20:22:50,043 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-09 20:22:50,054 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-09 20:22:50,054 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-02-09 20:22:50,055 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-02-09 20:22:50,055 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-09 20:22:50,060 - DEBUG - Requesting: http://10.0.0.33:8082/domain/31d0b242-908a-4dc4-83fa-cb50f11c1ca4 2026-02-09 20:22:50,067 - DEBUG - Requesting: http://10.0.0.33:8082/project/bfa9c0dc-1c9d-4d40-8bb4-c298630aa513 2026-02-09 20:22:50,110 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:50,119 - DEBUG - =>VN __public__ has no policy to be verified 2026-02-09 20:22:50,119 - DEBUG - Verifying the vn in opserver 2026-02-09 20:22:50,119 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.33 2026-02-09 20:22:50,119 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-02-09 20:22:50,124 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-02-09 20:22:50,124 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-02-09 20:22:50,124 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-09 20:22:50,138 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-09 20:22:50,152 - DEBUG - Do not have enough data to verify VN in agent 2026-02-09 20:22:50,152 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-09 20:22:50,166 - DEBUG - VRF ids for VN __public__: {} 2026-02-09 20:22:50,289 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-02-09 20:22:50,388 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-09 20:22:50,394 - DEBUG - Requesting: http://10.0.0.33:8082/domain/31d0b242-908a-4dc4-83fa-cb50f11c1ca4 2026-02-09 20:22:50,410 - DEBUG - Requesting: http://10.0.0.33:8082/project/bfa9c0dc-1c9d-4d40-8bb4-c298630aa513 2026-02-09 20:22:50,451 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:50,462 - DEBUG - Requesting: http://10.0.0.33:8082/floating-ip-pool/405ae0ba-c7a6-4a8a-bf08-a04d04f3d0a4 2026-02-09 20:22:50,469 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-09 20:22:50,473 - DEBUG - Requesting: http://10.0.0.33:8082/domain/31d0b242-908a-4dc4-83fa-cb50f11c1ca4 2026-02-09 20:22:50,480 - DEBUG - Requesting: http://10.0.0.33:8082/project/bfa9c0dc-1c9d-4d40-8bb4-c298630aa513 2026-02-09 20:22:50,519 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/562b4147-b17e-453c-83d2-e171af0cc0a1 2026-02-09 20:22:50,529 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-02-09 20:22:50,529 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-02-09 20:22:50,538 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-09 20:22:50,545 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-02-09 20:22:50,546 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-02-09 20:22:50,549 - DEBUG - Ingress ctest-nginx-ingress-22689029 not present 2026-02-09 20:22:50,549 - INFO - Creating Ingress ctest-nginx-ingress-22689029 2026-02-09 20:22:50,558 - INFO - Created Ingress ctest-nginx-ingress-22689029 2026-02-09 20:22:50,561 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-22689029 2026-02-09 20:22:53,569 - DEBUG - For Ingress ctest-nginx-ingress-22689029, Cluster IP: 10.47.255.249, External IPs ['13.130.178.252'] 2026-02-09 20:22:53,569 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-22689029 2026-02-09 20:22:53,570 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_LoadbalancerDatabaseList?x=79145141-ab06-4d9b-a88d-ed2fc118f371 2026-02-09 20:22:53,577 - INFO - Ingress ctest-nginx-ingress-22689029 with uuid 79145141-ab06-4d9b-a88d-ed2fc118f371 found in kube manager 2026-02-09 20:22:53,577 - INFO - Ingress ctest-nginx-ingress-22689029 verification passed 2026-02-09 20:22:53,582 - DEBUG - Pod ctest-busybox-pod-65740413 not present 2026-02-09 20:22:53,582 - INFO - Creating Pod ctest-busybox-pod-65740413 2026-02-09 20:22:53,641 - DEBUG - Pod : ctest-busybox-pod-65740413 UUID is 1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 2026-02-09 20:22:53,665 - DEBUG - Pod ctest-nginx-pod-43319799 not in running state.Currently in Pending 2026-02-09 20:22:58,679 - INFO - Pod ctest-nginx-pod-43319799 is in running state.Got IP 10.47.255.251 2026-02-09 20:22:58,695 - DEBUG - Pod ctest-nginx-pod-43319799 has vmi 184991ec-05f5-11f1-8620-fa163e686c0e 2026-02-09 20:22:58,695 - INFO - Verified pod ctest-nginx-pod-43319799 in contrail-api 2026-02-09 20:22:58,695 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-09 20:22:58,709 - DEBUG - VMI 184991ec-05f5-11f1-8620-fa163e686c0e is active in agent 10.0.0.19 2026-02-09 20:22:58,710 - INFO - Verified Pod ctest-nginx-pod-43319799 in agent 10.0.0.19 2026-02-09 20:22:58,710 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=afe8ff22-b452-4c21-b6d0-8f2f945969ed 2026-02-09 20:22:58,717 - INFO - Pod ctest-nginx-pod-43319799 with uuid afe8ff22-b452-4c21-b6d0-8f2f945969ed found in kube manager 2026-02-09 20:22:58,717 - INFO - Pod ctest-nginx-pod-43319799 verification passed 2026-02-09 20:22:58,830 - DEBUG - [Pod ctest-nginx-pod-43319799] Cmd: echo ctest-nginx-pod-43319799 > /usr/share/nginx/html/index.html, Output: 2026-02-09 20:22:58,838 - INFO - Pod ctest-nginx-pod-64627090 is in running state.Got IP 10.47.255.250 2026-02-09 20:22:58,893 - DEBUG - Pod ctest-nginx-pod-64627090 has vmi 18b1e620-05f5-11f1-8620-fa163e686c0e 2026-02-09 20:22:58,893 - INFO - Verified pod ctest-nginx-pod-64627090 in contrail-api 2026-02-09 20:22:58,893 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-09 20:22:58,904 - DEBUG - VMI 18b1e620-05f5-11f1-8620-fa163e686c0e is active in agent 10.0.0.19 2026-02-09 20:22:58,904 - INFO - Verified Pod ctest-nginx-pod-64627090 in agent 10.0.0.19 2026-02-09 20:22:58,905 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=2806cf5c-cfe0-40d4-aff1-4dd74e87be9c 2026-02-09 20:22:58,909 - INFO - Pod ctest-nginx-pod-64627090 with uuid 2806cf5c-cfe0-40d4-aff1-4dd74e87be9c found in kube manager 2026-02-09 20:22:58,909 - INFO - Pod ctest-nginx-pod-64627090 verification passed 2026-02-09 20:22:58,988 - DEBUG - [Pod ctest-nginx-pod-64627090] Cmd: echo ctest-nginx-pod-64627090 > /usr/share/nginx/html/index.html, Output: 2026-02-09 20:22:58,995 - DEBUG - Pod ctest-busybox-pod-65740413 not in running state.Currently in Pending 2026-02-09 20:23:04,007 - DEBUG - Pod ctest-busybox-pod-65740413 not in running state.Currently in Pending 2026-02-09 20:23:09,016 - INFO - Pod ctest-busybox-pod-65740413 is in running state.Got IP 10.47.255.248 2026-02-09 20:23:09,030 - DEBUG - Pod ctest-busybox-pod-65740413 has vmi 1d10e5f4-05f5-11f1-8620-fa163e686c0e 2026-02-09 20:23:09,030 - INFO - Verified pod ctest-busybox-pod-65740413 in contrail-api 2026-02-09 20:23:09,030 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-09 20:23:09,047 - DEBUG - VMI 1d10e5f4-05f5-11f1-8620-fa163e686c0e is active in agent 10.0.0.19 2026-02-09 20:23:09,047 - INFO - Verified Pod ctest-busybox-pod-65740413 in agent 10.0.0.19 2026-02-09 20:23:09,047 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 2026-02-09 20:23:09,052 - INFO - Pod ctest-busybox-pod-65740413 with uuid 1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 found in kube manager 2026-02-09 20:23:09,052 - INFO - Pod ctest-busybox-pod-65740413 verification passed 2026-02-09 20:23:09,153 - DEBUG - [Pod ctest-busybox-pod-65740413] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-64627090 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-09 20:23:09,153 - DEBUG - [Pod Pod ctest-busybox-pod-65740413] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-02-09 20:23:09,153 - DEBUG - [Pod Pod ctest-busybox-pod-65740413] Cmd output: ctest-nginx-pod-64627090 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-09 20:23:09,228 - DEBUG - [Pod ctest-busybox-pod-65740413] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-64627090 2026-02-09 20:23:09,228 - DEBUG - [Pod Pod ctest-busybox-pod-65740413] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-02-09 20:23:09,228 - DEBUG - [Pod Pod ctest-busybox-pod-65740413] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-64627090 2026-02-09 20:23:09,290 - DEBUG - [Pod ctest-busybox-pod-65740413] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-43319799 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-09 20:23:09,290 - DEBUG - [Pod Pod ctest-busybox-pod-65740413] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-02-09 20:23:09,290 - DEBUG - [Pod Pod ctest-busybox-pod-65740413] Cmd output: ctest-nginx-pod-43319799 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-09 20:23:09,290 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-43319799': 1, 'ctest-nginx-pod-64627090': 2} 2026-02-09 20:23:09,290 - WARNING - Nothing to delete parallely 2026-02-09 20:23:09,290 - INFO - Deleting pod default:ctest-busybox-pod-65740413 2026-02-09 20:23:09,308 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 2026-02-09 20:23:09,318 - WARNING - Pod uuid 1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 is still seen in agent 10.0.0.19 VM list 2026-02-09 20:23:14,319 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 2026-02-09 20:23:14,336 - WARNING - Pod uuid 1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 is still seen in agent 10.0.0.19 VM list 2026-02-09 20:23:19,336 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 2026-02-09 20:23:19,349 - WARNING - Pod uuid 1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 is still seen in agent 10.0.0.19 VM list 2026-02-09 20:23:24,349 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 2026-02-09 20:23:24,358 - WARNING - Pod uuid 1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 is still seen in agent 10.0.0.19 VM list 2026-02-09 20:23:29,359 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 2026-02-09 20:23:29,370 - WARNING - Pod uuid 1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 is still seen in agent 10.0.0.19 VM list 2026-02-09 20:23:34,371 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 2026-02-09 20:23:34,395 - WARNING - Pod uuid 1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 is still seen in agent 10.0.0.19 VM list 2026-02-09 20:23:39,397 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 2026-02-09 20:23:39,410 - WARNING - Pod uuid 1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 is still seen in agent 10.0.0.19 VM list 2026-02-09 20:23:44,411 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 2026-02-09 20:23:44,428 - DEBUG - Pod 1fc985e6-ae49-4a2e-9ee5-f5a9028e59a8 is not in agent 10.0.0.19 VM list 2026-02-09 20:23:44,428 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1d10e5f4-05f5-11f1-8620-fa163e686c0e 2026-02-09 20:23:44,449 - DEBUG - VMI 1d10e5f4-05f5-11f1-8620-fa163e686c0e is removed from agent 10.0.0.19 2026-02-09 20:23:44,449 - INFO - Verified that pod ctest-busybox-pod-65740413 is removed in agent 2026-02-09 20:23:44,449 - INFO - Deleting Ingress : ctest-nginx-ingress-22689029 2026-02-09 20:23:44,456 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_LoadbalancerDatabaseList?x=79145141-ab06-4d9b-a88d-ed2fc118f371 2026-02-09 20:23:44,460 - ERROR - Ingress ctest-nginx-ingress-22689029 with uuid 79145141-ab06-4d9b-a88d-ed2fc118f371 still found in kube manager 2026-02-09 20:23:45,461 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_LoadbalancerDatabaseList?x=79145141-ab06-4d9b-a88d-ed2fc118f371 2026-02-09 20:23:45,465 - INFO - Ingress ctest-nginx-ingress-22689029 with uuid 79145141-ab06-4d9b-a88d-ed2fc118f371 deleted successfully from kube manager 2026-02-09 20:23:45,466 - INFO - Verifications on Ingress %s cleanup passed 2026-02-09 20:23:45,466 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-02-09 20:23:45,560 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-09 20:23:45,568 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-02-09 20:23:45,568 - INFO - Deleting VN __public__ 2026-02-09 20:23:45,718 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/5f1719d3-ee5a-4241-83bb-a686d04e9308 2026-02-09 20:23:45,727 - DEBUG - Response Code: 404 2026-02-09 20:23:45,727 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-09 20:23:45,733 - DEBUG - Requesting: http://10.0.0.33:8082/domain/31d0b242-908a-4dc4-83fa-cb50f11c1ca4 2026-02-09 20:23:45,755 - DEBUG - Requesting: http://10.0.0.33:8082/project/bfa9c0dc-1c9d-4d40-8bb4-c298630aa513 2026-02-09 20:23:45,817 - INFO - Validated that VN __public__ is not found in API Server 2026-02-09 20:23:45,817 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-02-09 20:23:45,825 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-09 20:23:45,834 - DEBUG - VN __public__ is not present in Agent 10.0.0.19 2026-02-09 20:23:45,835 - INFO - Validated that VN __public__ is not in any agent 2026-02-09 20:23:45,835 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-09 20:23:45,853 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-09 20:23:45,858 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-09 20:23:45,863 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-02-09 20:23:45,863 - INFO - Deleting pod default:ctest-nginx-pod-64627090 2026-02-09 20:23:45,873 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=2806cf5c-cfe0-40d4-aff1-4dd74e87be9c 2026-02-09 20:23:45,882 - WARNING - Pod uuid 2806cf5c-cfe0-40d4-aff1-4dd74e87be9c is still seen in agent 10.0.0.19 VM list 2026-02-09 20:23:50,883 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=2806cf5c-cfe0-40d4-aff1-4dd74e87be9c 2026-02-09 20:23:50,890 - DEBUG - Pod 2806cf5c-cfe0-40d4-aff1-4dd74e87be9c is not in agent 10.0.0.19 VM list 2026-02-09 20:23:50,891 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:18b1e620-05f5-11f1-8620-fa163e686c0e 2026-02-09 20:23:50,896 - DEBUG - VMI 18b1e620-05f5-11f1-8620-fa163e686c0e is removed from agent 10.0.0.19 2026-02-09 20:23:50,896 - INFO - Verified that pod ctest-nginx-pod-64627090 is removed in agent 2026-02-09 20:23:50,897 - INFO - Deleting pod default:ctest-nginx-pod-43319799 2026-02-09 20:23:50,918 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=afe8ff22-b452-4c21-b6d0-8f2f945969ed 2026-02-09 20:23:50,929 - WARNING - Pod uuid afe8ff22-b452-4c21-b6d0-8f2f945969ed is still seen in agent 10.0.0.19 VM list 2026-02-09 20:23:55,929 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=afe8ff22-b452-4c21-b6d0-8f2f945969ed 2026-02-09 20:23:55,940 - DEBUG - Pod afe8ff22-b452-4c21-b6d0-8f2f945969ed is not in agent 10.0.0.19 VM list 2026-02-09 20:23:55,940 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:184991ec-05f5-11f1-8620-fa163e686c0e 2026-02-09 20:23:55,952 - DEBUG - VMI 184991ec-05f5-11f1-8620-fa163e686c0e is removed from agent 10.0.0.19 2026-02-09 20:23:55,952 - INFO - Verified that pod ctest-nginx-pod-43319799 is removed in agent 2026-02-09 20:23:55,952 - INFO - Deleting service : ctest-nginx-svc-95681849 2026-02-09 20:23:56,287 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-09 20:23:56,294 - DEBUG - No XMPP flaps were noticed during the test