2025-10-07 19:03:17,840 - DEBUG - [10.0.0.50]: Running cmd : hostname 2025-10-07 19:03:18,169 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1538-1 2025-10-07 19:03:18,169 - DEBUG - [10.0.0.50]: Running cmd : hostname -f 2025-10-07 19:03:18,235 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1538-1 2025-10-07 19:03:18,235 - DEBUG - [10.0.0.50]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-07 19:03:18,357 - DEBUG - Output : NAMES contrail_test_A99Y9hRpr contrail-kubernetes-master-kubemanager-1 analytics_database-nodemgr-1 analytics_database-query-engine-1 analytics_database-cassandra-1 analytics_snmp-snmp-collector-1 analytics_snmp-topology-1 analytics_snmp-nodemgr-1 analytics_alarm-alarm-gen-1 analytics_alarm-kafka-1 analytics_alarm-nodemgr-1 analytics-collector-1 analytics-nodemgr-1 analytics-api-1 redis-redis-1 webui-job-1 webui-web-1 control-named-1 control-dns-1 control-nodemgr-1 control-control-1 config_database-zookeeper-1 config_database-nodemgr-1 config_database-rabbitmq-1 config_database-cassandra-1 config_api-devicemgr-1 config_api-api-1 config_api-nodemgr-1 config_api-schema-1 config_api-svcmonitor-1 config_api-dnsmasq-1 2025-10-07 19:03:18,358 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-10-07 19:03:18,417 - DEBUG - Output : 127.0.0.1/8 10.0.0.50/24 252.50.0.1/8 10.164.103.1/24 172.17.0.1/16 2025-10-07 19:03:18,417 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.0.0.50 | head -n 1 | awk '{print $2}' 2025-10-07 19:03:18,464 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1538-1 2025-10-07 19:03:18,579 - DEBUG - [10.0.0.65]: Running cmd : hostname 2025-10-07 19:03:18,811 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1538-1 2025-10-07 19:03:18,812 - DEBUG - [10.0.0.65]: Running cmd : hostname -f 2025-10-07 19:03:18,846 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1538-1 2025-10-07 19:03:18,846 - DEBUG - [10.0.0.65]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-07 19:03:18,934 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2025-10-07 19:03:18,934 - DEBUG - [10.0.0.65]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-10-07 19:03:18,978 - DEBUG - Output : 10.0.0.65/24 2025-10-07 19:03:18,978 - DEBUG - [10.0.0.65]: Running cmd : getent hosts 10.0.0.65 | head -n 1 | awk '{print $2}' 2025-10-07 19:03:19,012 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1538-1 2025-10-07 19:03:19,073 - INFO - Using existing project ['default-domain', 'k8s-default'](133b2eeb-ba81-4f0d-ac78-0dafd604b718) 2025-10-07 19:03:19,130 - INFO - Using existing project ['default-domain', 'k8s-default'](133b2eeb-ba81-4f0d-ac78-0dafd604b718) 2025-10-07 19:03:19,231 - INFO - ================================================================================ 2025-10-07 19:03:19,231 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-10-07 19:03:19,231 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-10-07 19:03:19,411 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-07 19:03:19,417 - DEBUG - Nothing to compare xmpp stats {'10.0.0.65': {'10.0.0.50': '0'}} with 2025-10-07 19:03:19,417 - INFO - Initial checks done. Running the testcase now 2025-10-07 19:03:19,417 - INFO - 2025-10-07 19:03:19,421 - INFO - Namespace default already exists 2025-10-07 19:03:19,421 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-10-07 19:03:19,537 - DEBUG - Output : cluster_project={} 2025-10-07 19:03:19,580 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-10-07 19:03:19,580 - DEBUG - Project uuid in Contrail is 133b2eeb-ba81-4f0d-ac78-0dafd604b718 2025-10-07 19:03:19,580 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-10-07 19:03:19,707 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-10-07 19:03:19,707 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_NamespaceDatabaseList?namespace_uuid=edb07ba3-e167-495f-85bd-a68425927ddb 2025-10-07 19:03:19,711 - INFO - Namespace default with uuid edb07ba3-e167-495f-85bd-a68425927ddb found in kube manager 2025-10-07 19:03:19,711 - INFO - Namespace default verification passed 2025-10-07 19:03:19,714 - DEBUG - Service ctest-nginx-svc-09029588 not present 2025-10-07 19:03:19,715 - INFO - Creating service ctest-nginx-svc-09029588 2025-10-07 19:03:19,729 - INFO - Created Service ctest-nginx-svc-09029588 2025-10-07 19:03:19,736 - DEBUG - Pod ctest-nginx-pod-74514163 not present 2025-10-07 19:03:19,737 - INFO - Creating Pod ctest-nginx-pod-74514163 2025-10-07 19:03:19,785 - DEBUG - Pod : ctest-nginx-pod-74514163 UUID is 014d0986-844c-4a15-90aa-1304fa454a4c 2025-10-07 19:03:19,789 - DEBUG - Pod ctest-nginx-pod-59104158 not present 2025-10-07 19:03:19,789 - INFO - Creating Pod ctest-nginx-pod-59104158 2025-10-07 19:03:19,820 - DEBUG - Pod : ctest-nginx-pod-59104158 UUID is b6b91267-f537-4981-9dba-cc0c5be73001 2025-10-07 19:03:19,983 - INFO - Created VN __public__, UUID :0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:20,206 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2025-10-07 19:03:20,212 - DEBUG - Requesting: http://10.0.0.50:8082/domain/23eee8fe-fc16-4915-9bcf-393450324b88 2025-10-07 19:03:20,231 - DEBUG - Requesting: http://10.0.0.50:8082/project/133b2eeb-ba81-4f0d-ac78-0dafd604b718 2025-10-07 19:03:20,350 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:20,366 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:20,390 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a3f72ddb-dfd0-434a-ba0b-3f1ddbfebbdf 2025-10-07 19:03:20,397 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a3f72ddb-dfd0-434a-ba0b-3f1ddbfebbdf 2025-10-07 19:03:20,415 - DEBUG - Route Targets: [] 2025-10-07 19:03:20,415 - DEBUG - RT names not yet present for VN __public__ 2025-10-07 19:03:25,415 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2025-10-07 19:03:25,420 - DEBUG - Requesting: http://10.0.0.50:8082/domain/23eee8fe-fc16-4915-9bcf-393450324b88 2025-10-07 19:03:25,426 - DEBUG - Requesting: http://10.0.0.50:8082/project/133b2eeb-ba81-4f0d-ac78-0dafd604b718 2025-10-07 19:03:25,461 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:25,471 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:25,481 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a3f72ddb-dfd0-434a-ba0b-3f1ddbfebbdf 2025-10-07 19:03:25,495 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a3f72ddb-dfd0-434a-ba0b-3f1ddbfebbdf 2025-10-07 19:03:25,504 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/c04a015a-bb80-41d8-8f91-64ae8ceded10 2025-10-07 19:03:25,509 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-07 19:03:25,509 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:25,516 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a3f72ddb-dfd0-434a-ba0b-3f1ddbfebbdf 2025-10-07 19:03:25,527 - INFO - Verified VN network id 13 for VN 0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:25,528 - INFO - Verifications in API Server for VN __public__ passed 2025-10-07 19:03:25,528 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:25,535 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a3f72ddb-dfd0-434a-ba0b-3f1ddbfebbdf 2025-10-07 19:03:25,540 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a3f72ddb-dfd0-434a-ba0b-3f1ddbfebbdf 2025-10-07 19:03:25,545 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/c04a015a-bb80-41d8-8f91-64ae8ceded10 2025-10-07 19:03:25,550 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-07 19:03:25,555 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '733696710892996453', 'uuid-lslong': '12054013027295402815'}, 'enable': 'true', 'created': '2025-10-07T19:03:19', 'last-modified': '2025-10-07T19:03:20', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.466949'} 2025-10-07 19:03:25,555 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-07 19:03:25,564 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-07 19:03:25,564 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-10-07 19:03:25,564 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-10-07 19:03:25,565 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2025-10-07 19:03:25,570 - DEBUG - Requesting: http://10.0.0.50:8082/domain/23eee8fe-fc16-4915-9bcf-393450324b88 2025-10-07 19:03:25,576 - DEBUG - Requesting: http://10.0.0.50:8082/project/133b2eeb-ba81-4f0d-ac78-0dafd604b718 2025-10-07 19:03:25,616 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:25,626 - DEBUG - =>VN __public__ has no policy to be verified 2025-10-07 19:03:25,626 - DEBUG - Verifying the vn in opserver 2025-10-07 19:03:25,626 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.50 2025-10-07 19:03:25,626 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2025-10-07 19:03:25,631 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-10-07 19:03:25,632 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-10-07 19:03:25,632 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-07 19:03:25,642 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-07 19:03:25,647 - DEBUG - Do not have enough data to verify VN in agent 2025-10-07 19:03:25,647 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-07 19:03:25,654 - DEBUG - VRF ids for VN __public__: {} 2025-10-07 19:03:25,753 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-10-07 19:03:25,844 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2025-10-07 19:03:25,849 - DEBUG - Requesting: http://10.0.0.50:8082/domain/23eee8fe-fc16-4915-9bcf-393450324b88 2025-10-07 19:03:25,855 - DEBUG - Requesting: http://10.0.0.50:8082/project/133b2eeb-ba81-4f0d-ac78-0dafd604b718 2025-10-07 19:03:25,883 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:25,894 - DEBUG - Requesting: http://10.0.0.50:8082/floating-ip-pool/49ee0752-a3d6-4ad4-a361-f8bcf711d756 2025-10-07 19:03:25,899 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2025-10-07 19:03:25,903 - DEBUG - Requesting: http://10.0.0.50:8082/domain/23eee8fe-fc16-4915-9bcf-393450324b88 2025-10-07 19:03:25,910 - DEBUG - Requesting: http://10.0.0.50:8082/project/133b2eeb-ba81-4f0d-ac78-0dafd604b718 2025-10-07 19:03:25,935 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/0a2e9d45-71a6-4765-a748-74fbc1df9b3f 2025-10-07 19:03:25,945 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-10-07 19:03:25,945 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-10-07 19:03:25,952 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-07 19:03:25,956 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-10-07 19:03:25,956 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-10-07 19:03:25,959 - DEBUG - Ingress ctest-nginx-ingress-48627746 not present 2025-10-07 19:03:25,960 - INFO - Creating Ingress ctest-nginx-ingress-48627746 2025-10-07 19:03:25,966 - INFO - Created Ingress ctest-nginx-ingress-48627746 2025-10-07 19:03:25,969 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-48627746 2025-10-07 19:03:28,974 - DEBUG - For Ingress ctest-nginx-ingress-48627746, Cluster IP: 10.47.255.249, External IPs ['134.65.195.188'] 2025-10-07 19:03:28,974 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-48627746 2025-10-07 19:03:28,974 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=c4b20348-ce19-48d7-bc64-57a73800a8b2 2025-10-07 19:03:28,978 - INFO - Ingress ctest-nginx-ingress-48627746 with uuid c4b20348-ce19-48d7-bc64-57a73800a8b2 found in kube manager 2025-10-07 19:03:28,978 - INFO - Ingress ctest-nginx-ingress-48627746 verification passed 2025-10-07 19:03:28,981 - DEBUG - Pod ctest-busybox-pod-01292848 not present 2025-10-07 19:03:28,981 - INFO - Creating Pod ctest-busybox-pod-01292848 2025-10-07 19:03:29,021 - DEBUG - Pod : ctest-busybox-pod-01292848 UUID is e9a8979c-02c8-46f8-9eb5-26b163a89afc 2025-10-07 19:03:29,027 - DEBUG - Pod ctest-nginx-pod-74514163 not in running state.Currently in Pending 2025-10-07 19:03:34,063 - DEBUG - Pod ctest-nginx-pod-74514163 not in running state.Currently in Pending 2025-10-07 19:03:39,070 - INFO - Pod ctest-nginx-pod-74514163 is in running state.Got IP 10.47.255.251 2025-10-07 19:03:39,086 - DEBUG - Pod ctest-nginx-pod-74514163 has vmi 4abcbc5a-a3b0-11f0-a578-fa163e9ddeb1 2025-10-07 19:03:39,087 - INFO - Verified pod ctest-nginx-pod-74514163 in contrail-api 2025-10-07 19:03:39,087 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-07 19:03:39,104 - DEBUG - VMI 4abcbc5a-a3b0-11f0-a578-fa163e9ddeb1 is active in agent 10.0.0.65 2025-10-07 19:03:39,105 - INFO - Verified Pod ctest-nginx-pod-74514163 in agent 10.0.0.65 2025-10-07 19:03:39,105 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=014d0986-844c-4a15-90aa-1304fa454a4c 2025-10-07 19:03:39,116 - INFO - Pod ctest-nginx-pod-74514163 with uuid 014d0986-844c-4a15-90aa-1304fa454a4c found in kube manager 2025-10-07 19:03:39,116 - INFO - Pod ctest-nginx-pod-74514163 verification passed 2025-10-07 19:03:39,185 - DEBUG - [Pod ctest-nginx-pod-74514163] Cmd: echo ctest-nginx-pod-74514163 > /usr/share/nginx/html/index.html, Output: 2025-10-07 19:03:39,192 - INFO - Pod ctest-nginx-pod-59104158 is in running state.Got IP 10.47.255.250 2025-10-07 19:03:39,203 - DEBUG - Pod ctest-nginx-pod-59104158 has vmi 4b200490-a3b0-11f0-a578-fa163e9ddeb1 2025-10-07 19:03:39,203 - INFO - Verified pod ctest-nginx-pod-59104158 in contrail-api 2025-10-07 19:03:39,203 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-07 19:03:39,216 - DEBUG - VMI 4b200490-a3b0-11f0-a578-fa163e9ddeb1 is active in agent 10.0.0.65 2025-10-07 19:03:39,216 - INFO - Verified Pod ctest-nginx-pod-59104158 in agent 10.0.0.65 2025-10-07 19:03:39,216 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=b6b91267-f537-4981-9dba-cc0c5be73001 2025-10-07 19:03:39,219 - INFO - Pod ctest-nginx-pod-59104158 with uuid b6b91267-f537-4981-9dba-cc0c5be73001 found in kube manager 2025-10-07 19:03:39,220 - INFO - Pod ctest-nginx-pod-59104158 verification passed 2025-10-07 19:03:39,273 - DEBUG - [Pod ctest-nginx-pod-59104158] Cmd: echo ctest-nginx-pod-59104158 > /usr/share/nginx/html/index.html, Output: 2025-10-07 19:03:39,279 - DEBUG - Pod ctest-busybox-pod-01292848 not in running state.Currently in Pending 2025-10-07 19:03:44,286 - INFO - Pod ctest-busybox-pod-01292848 is in running state.Got IP 10.47.255.248 2025-10-07 19:03:44,342 - DEBUG - Pod ctest-busybox-pod-01292848 has vmi 4f7e8764-a3b0-11f0-a578-fa163e9ddeb1 2025-10-07 19:03:44,342 - INFO - Verified pod ctest-busybox-pod-01292848 in contrail-api 2025-10-07 19:03:44,342 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-07 19:03:44,351 - DEBUG - VMI 4f7e8764-a3b0-11f0-a578-fa163e9ddeb1 is active in agent 10.0.0.65 2025-10-07 19:03:44,351 - INFO - Verified Pod ctest-busybox-pod-01292848 in agent 10.0.0.65 2025-10-07 19:03:44,352 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=e9a8979c-02c8-46f8-9eb5-26b163a89afc 2025-10-07 19:03:44,355 - INFO - Pod ctest-busybox-pod-01292848 with uuid e9a8979c-02c8-46f8-9eb5-26b163a89afc found in kube manager 2025-10-07 19:03:44,355 - INFO - Pod ctest-busybox-pod-01292848 verification passed 2025-10-07 19:03:44,413 - DEBUG - [Pod ctest-busybox-pod-01292848] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-59104158 2025-10-07 19:03:44,413 - DEBUG - [Pod Pod ctest-busybox-pod-01292848] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2025-10-07 19:03:44,413 - DEBUG - [Pod Pod ctest-busybox-pod-01292848] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-59104158 2025-10-07 19:03:44,463 - DEBUG - [Pod ctest-busybox-pod-01292848] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-74514163 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-07 19:03:44,463 - DEBUG - [Pod Pod ctest-busybox-pod-01292848] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2025-10-07 19:03:44,463 - DEBUG - [Pod Pod ctest-busybox-pod-01292848] Cmd output: ctest-nginx-pod-74514163 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-07 19:03:44,463 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-74514163': 1, 'ctest-nginx-pod-59104158': 1} 2025-10-07 19:03:44,463 - WARNING - Nothing to delete parallely 2025-10-07 19:03:44,464 - INFO - Deleting pod default:ctest-busybox-pod-01292848 2025-10-07 19:03:44,480 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e9a8979c-02c8-46f8-9eb5-26b163a89afc 2025-10-07 19:03:44,485 - WARNING - Pod uuid e9a8979c-02c8-46f8-9eb5-26b163a89afc is still seen in agent 10.0.0.65 VM list 2025-10-07 19:03:49,486 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e9a8979c-02c8-46f8-9eb5-26b163a89afc 2025-10-07 19:03:49,492 - WARNING - Pod uuid e9a8979c-02c8-46f8-9eb5-26b163a89afc is still seen in agent 10.0.0.65 VM list 2025-10-07 19:03:54,493 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e9a8979c-02c8-46f8-9eb5-26b163a89afc 2025-10-07 19:03:54,500 - WARNING - Pod uuid e9a8979c-02c8-46f8-9eb5-26b163a89afc is still seen in agent 10.0.0.65 VM list 2025-10-07 19:03:59,500 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e9a8979c-02c8-46f8-9eb5-26b163a89afc 2025-10-07 19:03:59,508 - WARNING - Pod uuid e9a8979c-02c8-46f8-9eb5-26b163a89afc is still seen in agent 10.0.0.65 VM list 2025-10-07 19:04:04,509 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e9a8979c-02c8-46f8-9eb5-26b163a89afc 2025-10-07 19:04:04,518 - WARNING - Pod uuid e9a8979c-02c8-46f8-9eb5-26b163a89afc is still seen in agent 10.0.0.65 VM list 2025-10-07 19:04:09,519 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e9a8979c-02c8-46f8-9eb5-26b163a89afc 2025-10-07 19:04:09,528 - WARNING - Pod uuid e9a8979c-02c8-46f8-9eb5-26b163a89afc is still seen in agent 10.0.0.65 VM list 2025-10-07 19:04:14,529 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e9a8979c-02c8-46f8-9eb5-26b163a89afc 2025-10-07 19:04:14,538 - WARNING - Pod uuid e9a8979c-02c8-46f8-9eb5-26b163a89afc is still seen in agent 10.0.0.65 VM list 2025-10-07 19:04:19,539 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e9a8979c-02c8-46f8-9eb5-26b163a89afc 2025-10-07 19:04:19,545 - DEBUG - Pod e9a8979c-02c8-46f8-9eb5-26b163a89afc is not in agent 10.0.0.65 VM list 2025-10-07 19:04:19,545 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4f7e8764-a3b0-11f0-a578-fa163e9ddeb1 2025-10-07 19:04:19,549 - DEBUG - VMI 4f7e8764-a3b0-11f0-a578-fa163e9ddeb1 is removed from agent 10.0.0.65 2025-10-07 19:04:19,549 - INFO - Verified that pod ctest-busybox-pod-01292848 is removed in agent 2025-10-07 19:04:19,550 - INFO - Deleting Ingress : ctest-nginx-ingress-48627746 2025-10-07 19:04:19,556 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=c4b20348-ce19-48d7-bc64-57a73800a8b2 2025-10-07 19:04:19,562 - ERROR - Ingress ctest-nginx-ingress-48627746 with uuid c4b20348-ce19-48d7-bc64-57a73800a8b2 still found in kube manager 2025-10-07 19:04:20,569 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=c4b20348-ce19-48d7-bc64-57a73800a8b2 2025-10-07 19:04:20,573 - INFO - Ingress ctest-nginx-ingress-48627746 with uuid c4b20348-ce19-48d7-bc64-57a73800a8b2 deleted successfully from kube manager 2025-10-07 19:04:20,574 - INFO - Verifications on Ingress %s cleanup passed 2025-10-07 19:04:20,574 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-10-07 19:04:20,673 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-07 19:04:20,678 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-10-07 19:04:20,678 - INFO - Deleting VN __public__ 2025-10-07 19:04:20,795 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a3f72ddb-dfd0-434a-ba0b-3f1ddbfebbdf 2025-10-07 19:04:20,800 - DEBUG - Response Code: 404 2025-10-07 19:04:20,800 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2025-10-07 19:04:20,806 - DEBUG - Requesting: http://10.0.0.50:8082/domain/23eee8fe-fc16-4915-9bcf-393450324b88 2025-10-07 19:04:20,812 - DEBUG - Requesting: http://10.0.0.50:8082/project/133b2eeb-ba81-4f0d-ac78-0dafd604b718 2025-10-07 19:04:20,848 - INFO - Validated that VN __public__ is not found in API Server 2025-10-07 19:04:20,848 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-10-07 19:04:20,856 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-07 19:04:20,867 - DEBUG - VN __public__ is not present in Agent 10.0.0.65 2025-10-07 19:04:20,867 - INFO - Validated that VN __public__ is not in any agent 2025-10-07 19:04:20,868 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-07 19:04:20,875 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-07 19:04:20,880 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-07 19:04:20,884 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-10-07 19:04:20,884 - INFO - Deleting pod default:ctest-nginx-pod-59104158 2025-10-07 19:04:20,897 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=b6b91267-f537-4981-9dba-cc0c5be73001 2025-10-07 19:04:20,910 - WARNING - Pod uuid b6b91267-f537-4981-9dba-cc0c5be73001 is still seen in agent 10.0.0.65 VM list 2025-10-07 19:04:25,911 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=b6b91267-f537-4981-9dba-cc0c5be73001 2025-10-07 19:04:25,916 - DEBUG - Pod b6b91267-f537-4981-9dba-cc0c5be73001 is not in agent 10.0.0.65 VM list 2025-10-07 19:04:25,917 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4b200490-a3b0-11f0-a578-fa163e9ddeb1 2025-10-07 19:04:25,921 - DEBUG - VMI 4b200490-a3b0-11f0-a578-fa163e9ddeb1 is removed from agent 10.0.0.65 2025-10-07 19:04:25,921 - INFO - Verified that pod ctest-nginx-pod-59104158 is removed in agent 2025-10-07 19:04:25,922 - INFO - Deleting pod default:ctest-nginx-pod-74514163 2025-10-07 19:04:25,950 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=014d0986-844c-4a15-90aa-1304fa454a4c 2025-10-07 19:04:25,956 - WARNING - Pod uuid 014d0986-844c-4a15-90aa-1304fa454a4c is still seen in agent 10.0.0.65 VM list 2025-10-07 19:04:30,957 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=014d0986-844c-4a15-90aa-1304fa454a4c 2025-10-07 19:04:30,963 - DEBUG - Pod 014d0986-844c-4a15-90aa-1304fa454a4c is not in agent 10.0.0.65 VM list 2025-10-07 19:04:30,963 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4abcbc5a-a3b0-11f0-a578-fa163e9ddeb1 2025-10-07 19:04:30,968 - DEBUG - VMI 4abcbc5a-a3b0-11f0-a578-fa163e9ddeb1 is removed from agent 10.0.0.65 2025-10-07 19:04:30,968 - INFO - Verified that pod ctest-nginx-pod-74514163 is removed in agent 2025-10-07 19:04:30,968 - INFO - Deleting service : ctest-nginx-svc-09029588 2025-10-07 19:04:31,153 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-07 19:04:31,160 - DEBUG - No XMPP flaps were noticed during the test