2025-10-31 19:37:40,541 - DEBUG - [10.0.0.65]: Running cmd : hostname 2025-10-31 19:37:40,816 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-406-1.novalocal 2025-10-31 19:37:40,816 - DEBUG - [10.0.0.65]: Running cmd : hostname -f 2025-10-31 19:37:40,949 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-406-1.novalocal 2025-10-31 19:37:40,949 - DEBUG - [10.0.0.65]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-31 19:37:41,137 - DEBUG - Output : NAMES contrail_test_wgz9KAj8U vrouter_vrouter-agent_1 vrouter_nodemgr_1 rsyslogd_rsyslogd_1 kubemanager_k8s_kubemanager_1 analytics_snmp_nodemgr_1 analytics_snmp_snmp-collector_1 analytics_snmp_topology_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_collector_1 analytics_api_1 analytics_database_nodemgr_1 analytics_database_query-engine_1 analytics_database_cassandra_1 control_named_1 control_dns_1 control_nodemgr_1 control_control_1 webui_job_1 webui_web_1 config_devicemgr_1 config_schema_1 config_nodemgr_1 config_dnsmasq_1 config_svcmonitor_1 config_api_1 config_database_nodemgr_1 config_database_rabbitmq_1 config_database_zookeeper_1 config_database_cassandra_1 redis_redis_1 2025-10-31 19:37:41,137 - DEBUG - [10.0.0.65]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-10-31 19:37:41,265 - DEBUG - Output : 127.0.0.1/8 10.0.0.65/24 10.20.0.17/24 172.17.0.1/16 10.20.0.17/24 2025-10-31 19:37:41,265 - DEBUG - [10.0.0.65]: Running cmd : getent hosts 10.0.0.65 | head -n 1 | awk '{print $2}' 2025-10-31 19:37:41,398 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-406-1.novalocal 2025-10-31 19:37:41,398 - DEBUG - [10.0.0.65]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-10-31 19:37:41,528 - DEBUG - Output : 10.20.0.17/24 2025-10-31 19:37:41,528 - DEBUG - [10.0.0.65]: Running cmd : getent hosts 10.20.0.17 | head -n 1 | awk '{print $2}' 2025-10-31 19:37:41,661 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-406-1.novalocal 2025-10-31 19:37:41,990 - INFO - Using existing project ['default-domain', 'k8s-default'](7c8b4bb8-79bc-439d-b45e-126cb331f18b) 2025-10-31 19:37:42,079 - INFO - Using existing project ['default-domain', 'k8s-default'](7c8b4bb8-79bc-439d-b45e-126cb331f18b) 2025-10-31 19:37:42,265 - INFO - ================================================================================ 2025-10-31 19:37:42,265 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-10-31 19:37:42,266 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-10-31 19:37:42,597 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-31 19:37:42,603 - DEBUG - Nothing to compare xmpp stats {'10.0.0.65': {'10.20.0.17': '0'}} with 2025-10-31 19:37:42,604 - INFO - Initial checks done. Running the testcase now 2025-10-31 19:37:42,604 - INFO - 2025-10-31 19:37:42,649 - INFO - Namespace default already exists 2025-10-31 19:37:42,650 - DEBUG - [10.0.0.65]: Running cmd : docker exec --privileged -it kubemanager_k8s_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-10-31 19:37:42,808 - DEBUG - Output : cluster_project={} 2025-10-31 19:37:42,876 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-10-31 19:37:42,876 - DEBUG - Project uuid in Contrail is 7c8b4bb8-79bc-439d-b45e-126cb331f18b 2025-10-31 19:37:42,876 - DEBUG - [10.0.0.65]: Running cmd : docker exec --privileged -it kubemanager_k8s_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-10-31 19:37:43,062 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-10-31 19:37:43,062 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_NamespaceDatabaseList?namespace_uuid=3310fb18-5dd2-401f-8113-11a7d446959b 2025-10-31 19:37:43,068 - INFO - Namespace default with uuid 3310fb18-5dd2-401f-8113-11a7d446959b found in kube manager 2025-10-31 19:37:43,068 - INFO - Namespace default verification passed 2025-10-31 19:37:43,071 - DEBUG - Service ctest-nginx-svc-82495844 not present 2025-10-31 19:37:43,072 - INFO - Creating service ctest-nginx-svc-82495844 2025-10-31 19:37:43,086 - INFO - Created Service ctest-nginx-svc-82495844 2025-10-31 19:37:43,096 - DEBUG - Pod ctest-nginx-pod-33311041 not present 2025-10-31 19:37:43,097 - INFO - Creating Pod ctest-nginx-pod-33311041 2025-10-31 19:37:43,151 - DEBUG - Pod : ctest-nginx-pod-33311041 UUID is c1e52dea-af8b-4daf-a426-cd6f23bcafdf 2025-10-31 19:37:43,168 - DEBUG - Pod ctest-nginx-pod-61377039 not present 2025-10-31 19:37:43,169 - INFO - Creating Pod ctest-nginx-pod-61377039 2025-10-31 19:37:43,218 - DEBUG - Pod : ctest-nginx-pod-61377039 UUID is 79ba52f3-6a9d-41cf-abe2-1cc874c024e7 2025-10-31 19:37:43,363 - INFO - Created VN __public__, UUID :44740ff8-e6b8-4a32-aa68-bdb6644c4519 2025-10-31 19:37:43,525 - DEBUG - Requesting: http://10.0.0.65:8082/domains 2025-10-31 19:37:43,530 - DEBUG - Requesting: http://10.0.0.65:8082/domain/252dd1eb-b22f-44c7-a25c-bbe9a34ef8f6 2025-10-31 19:37:43,546 - DEBUG - Requesting: http://10.0.0.65:8082/project/7c8b4bb8-79bc-439d-b45e-126cb331f18b 2025-10-31 19:37:43,585 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/44740ff8-e6b8-4a32-aa68-bdb6644c4519 2025-10-31 19:37:43,597 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/44740ff8-e6b8-4a32-aa68-bdb6644c4519 2025-10-31 19:37:43,607 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/c699f5b3-3982-4a03-9172-ef77333a1ae7 2025-10-31 19:37:43,613 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/c699f5b3-3982-4a03-9172-ef77333a1ae7 2025-10-31 19:37:43,620 - DEBUG - Requesting: http://10.0.0.65:8082/route-target/6b496d32-415d-4e1d-8dfa-a64dd364a126 2025-10-31 19:37:43,626 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-31 19:37:43,626 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/44740ff8-e6b8-4a32-aa68-bdb6644c4519 2025-10-31 19:37:43,636 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/c699f5b3-3982-4a03-9172-ef77333a1ae7 2025-10-31 19:37:43,649 - INFO - Verified VN network id 15 for VN 44740ff8-e6b8-4a32-aa68-bdb6644c4519 2025-10-31 19:37:43,649 - INFO - Verifications in API Server for VN __public__ passed 2025-10-31 19:37:43,649 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/44740ff8-e6b8-4a32-aa68-bdb6644c4519 2025-10-31 19:37:43,658 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/c699f5b3-3982-4a03-9172-ef77333a1ae7 2025-10-31 19:37:43,663 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/c699f5b3-3982-4a03-9172-ef77333a1ae7 2025-10-31 19:37:43,667 - DEBUG - Requesting: http://10.0.0.65:8082/route-target/6b496d32-415d-4e1d-8dfa-a64dd364a126 2025-10-31 19:37:43,671 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-31 19:37:43,677 - DEBUG - Control-node 10.0.0.65 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4932585053574679090', 'uuid-lslong': '12279272975090074905'}, 'enable': 'true', 'created': '2025-10-31T19:37:43', 'last-modified': '2025-10-31T19:37:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.201870'} 2025-10-31 19:37:43,677 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-31 19:37:43,685 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-31 19:37:43,685 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-10-31 19:37:43,687 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-10-31 19:37:43,687 - DEBUG - Requesting: http://10.0.0.65:8082/domains 2025-10-31 19:37:43,691 - DEBUG - Requesting: http://10.0.0.65:8082/domain/252dd1eb-b22f-44c7-a25c-bbe9a34ef8f6 2025-10-31 19:37:43,705 - DEBUG - Requesting: http://10.0.0.65:8082/project/7c8b4bb8-79bc-439d-b45e-126cb331f18b 2025-10-31 19:37:43,751 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/44740ff8-e6b8-4a32-aa68-bdb6644c4519 2025-10-31 19:37:43,758 - DEBUG - =>VN __public__ has no policy to be verified 2025-10-31 19:37:43,758 - DEBUG - Verifying the vn in opserver 2025-10-31 19:37:43,758 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.65 2025-10-31 19:37:43,759 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-networks 2025-10-31 19:37:43,763 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-10-31 19:37:43,763 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-10-31 19:37:43,763 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-31 19:37:43,769 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-31 19:37:43,774 - DEBUG - Do not have enough data to verify VN in agent 2025-10-31 19:37:43,774 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-31 19:37:43,779 - DEBUG - VRF ids for VN __public__: {} 2025-10-31 19:37:43,834 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-10-31 19:37:43,907 - DEBUG - Requesting: http://10.0.0.65:8082/domains 2025-10-31 19:37:43,913 - DEBUG - Requesting: http://10.0.0.65:8082/domain/252dd1eb-b22f-44c7-a25c-bbe9a34ef8f6 2025-10-31 19:37:43,928 - DEBUG - Requesting: http://10.0.0.65:8082/project/7c8b4bb8-79bc-439d-b45e-126cb331f18b 2025-10-31 19:37:43,975 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/44740ff8-e6b8-4a32-aa68-bdb6644c4519 2025-10-31 19:37:43,988 - DEBUG - Requesting: http://10.0.0.65:8082/floating-ip-pool/90e4f4d1-0f62-463b-9fcb-2e59f8158027 2025-10-31 19:37:43,994 - DEBUG - Requesting: http://10.0.0.65:8082/domains 2025-10-31 19:37:43,998 - DEBUG - Requesting: http://10.0.0.65:8082/domain/252dd1eb-b22f-44c7-a25c-bbe9a34ef8f6 2025-10-31 19:37:44,006 - DEBUG - Requesting: http://10.0.0.65:8082/project/7c8b4bb8-79bc-439d-b45e-126cb331f18b 2025-10-31 19:37:44,057 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/44740ff8-e6b8-4a32-aa68-bdb6644c4519 2025-10-31 19:37:44,066 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-10-31 19:37:44,066 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-10-31 19:37:44,074 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-31 19:37:44,080 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-10-31 19:37:44,080 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-10-31 19:37:44,083 - DEBUG - Ingress ctest-nginx-ingress-49473261 not present 2025-10-31 19:37:44,084 - INFO - Creating Ingress ctest-nginx-ingress-49473261 2025-10-31 19:37:44,092 - INFO - Created Ingress ctest-nginx-ingress-49473261 2025-10-31 19:37:44,097 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-49473261 2025-10-31 19:37:47,102 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-49473261 2025-10-31 19:37:50,108 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-49473261 2025-10-31 19:37:53,114 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-49473261 2025-10-31 19:37:56,120 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-49473261 2025-10-31 19:37:59,126 - DEBUG - For Ingress ctest-nginx-ingress-49473261, Cluster IP: 10.47.255.246, External IPs ['45.198.138.188'] 2025-10-31 19:37:59,126 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-49473261 2025-10-31 19:37:59,126 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:37:59,135 - INFO - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 found in kube manager 2025-10-31 19:37:59,135 - INFO - Ingress ctest-nginx-ingress-49473261 verification passed 2025-10-31 19:37:59,138 - DEBUG - Pod ctest-busybox-pod-96756663 not present 2025-10-31 19:37:59,139 - INFO - Creating Pod ctest-busybox-pod-96756663 2025-10-31 19:37:59,163 - DEBUG - Pod : ctest-busybox-pod-96756663 UUID is f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 2025-10-31 19:37:59,170 - DEBUG - Pod ctest-nginx-pod-33311041 not in running state.Currently in Pending 2025-10-31 19:38:04,179 - INFO - Pod ctest-nginx-pod-33311041 is in running state.Got IP 10.47.255.248 2025-10-31 19:38:04,193 - DEBUG - Pod ctest-nginx-pod-33311041 has vmi 13fe8c40-b691-11f0-8cce-fa163e1b5142 2025-10-31 19:38:04,193 - INFO - Verified pod ctest-nginx-pod-33311041 in contrail-api 2025-10-31 19:38:04,194 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-31 19:38:04,205 - DEBUG - VMI 13fe8c40-b691-11f0-8cce-fa163e1b5142 is active in agent 10.0.0.65 2025-10-31 19:38:04,205 - INFO - Verified Pod ctest-nginx-pod-33311041 in agent 10.0.0.65 2025-10-31 19:38:04,205 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_VirtualMachineDatabaseList?x=c1e52dea-af8b-4daf-a426-cd6f23bcafdf 2025-10-31 19:38:04,212 - INFO - Pod ctest-nginx-pod-33311041 with uuid c1e52dea-af8b-4daf-a426-cd6f23bcafdf found in kube manager 2025-10-31 19:38:04,213 - INFO - Pod ctest-nginx-pod-33311041 verification passed 2025-10-31 19:38:04,323 - DEBUG - [Pod ctest-nginx-pod-33311041] Cmd: echo ctest-nginx-pod-33311041 > /usr/share/nginx/html/index.html, Output: 2025-10-31 19:38:04,331 - INFO - Pod ctest-nginx-pod-61377039 is in running state.Got IP 10.47.255.247 2025-10-31 19:38:04,349 - DEBUG - Pod ctest-nginx-pod-61377039 has vmi 16140adc-b691-11f0-8cce-fa163e1b5142 2025-10-31 19:38:04,349 - INFO - Verified pod ctest-nginx-pod-61377039 in contrail-api 2025-10-31 19:38:04,349 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-31 19:38:04,358 - DEBUG - VMI 16140adc-b691-11f0-8cce-fa163e1b5142 is active in agent 10.0.0.65 2025-10-31 19:38:04,359 - INFO - Verified Pod ctest-nginx-pod-61377039 in agent 10.0.0.65 2025-10-31 19:38:04,359 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_VirtualMachineDatabaseList?x=79ba52f3-6a9d-41cf-abe2-1cc874c024e7 2025-10-31 19:38:04,363 - INFO - Pod ctest-nginx-pod-61377039 with uuid 79ba52f3-6a9d-41cf-abe2-1cc874c024e7 found in kube manager 2025-10-31 19:38:04,363 - INFO - Pod ctest-nginx-pod-61377039 verification passed 2025-10-31 19:38:04,429 - DEBUG - [Pod ctest-nginx-pod-61377039] Cmd: echo ctest-nginx-pod-61377039 > /usr/share/nginx/html/index.html, Output: 2025-10-31 19:38:04,437 - DEBUG - Pod ctest-busybox-pod-96756663 not in running state.Currently in Pending 2025-10-31 19:38:09,446 - DEBUG - Pod ctest-busybox-pod-96756663 not in running state.Currently in Pending 2025-10-31 19:38:14,456 - DEBUG - Pod ctest-busybox-pod-96756663 not in running state.Currently in Pending 2025-10-31 19:38:19,467 - DEBUG - Pod ctest-busybox-pod-96756663 not in running state.Currently in Pending 2025-10-31 19:38:24,478 - INFO - Pod ctest-busybox-pod-96756663 is in running state.Got IP 10.47.255.245 2025-10-31 19:38:24,494 - DEBUG - Pod ctest-busybox-pod-96756663 has vmi 21876bfc-b691-11f0-8cce-fa163e1b5142 2025-10-31 19:38:24,494 - INFO - Verified pod ctest-busybox-pod-96756663 in contrail-api 2025-10-31 19:38:24,494 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-31 19:38:24,505 - DEBUG - VMI 21876bfc-b691-11f0-8cce-fa163e1b5142 is active in agent 10.0.0.65 2025-10-31 19:38:24,505 - INFO - Verified Pod ctest-busybox-pod-96756663 in agent 10.0.0.65 2025-10-31 19:38:24,505 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_VirtualMachineDatabaseList?x=f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 2025-10-31 19:38:24,509 - INFO - Pod ctest-busybox-pod-96756663 with uuid f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 found in kube manager 2025-10-31 19:38:24,509 - INFO - Pod ctest-busybox-pod-96756663 verification passed 2025-10-31 19:38:24,562 - DEBUG - [Pod ctest-busybox-pod-96756663] Cmd: wget http://10.47.255.246:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.246:80 (10.47.255.246:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-33311041 2025-10-31 19:38:24,562 - DEBUG - [Pod Pod ctest-busybox-pod-96756663] Cmd wget http://10.47.255.246:80/ -O - -T 5 -t 1 passed 2025-10-31 19:38:24,562 - DEBUG - [Pod Pod ctest-busybox-pod-96756663] Cmd output: Connecting to 10.47.255.246:80 (10.47.255.246:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-33311041 2025-10-31 19:38:24,612 - DEBUG - [Pod ctest-busybox-pod-96756663] Cmd: wget http://10.47.255.246:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.246:80 (10.47.255.246:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-61377039 2025-10-31 19:38:24,612 - DEBUG - [Pod Pod ctest-busybox-pod-96756663] Cmd wget http://10.47.255.246:80/ -O - -T 5 -t 1 passed 2025-10-31 19:38:24,613 - DEBUG - [Pod Pod ctest-busybox-pod-96756663] Cmd output: Connecting to 10.47.255.246:80 (10.47.255.246:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-61377039 2025-10-31 19:38:24,613 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-33311041': 1, 'ctest-nginx-pod-61377039': 1} 2025-10-31 19:38:24,613 - WARNING - Nothing to delete parallely 2025-10-31 19:38:24,613 - INFO - Deleting pod default:ctest-busybox-pod-96756663 2025-10-31 19:38:24,631 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 2025-10-31 19:38:24,637 - WARNING - Pod uuid f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 is still seen in agent 10.0.0.65 VM list 2025-10-31 19:38:29,638 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 2025-10-31 19:38:29,643 - WARNING - Pod uuid f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 is still seen in agent 10.0.0.65 VM list 2025-10-31 19:38:34,644 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 2025-10-31 19:38:34,650 - WARNING - Pod uuid f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 is still seen in agent 10.0.0.65 VM list 2025-10-31 19:38:39,651 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 2025-10-31 19:38:39,657 - WARNING - Pod uuid f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 is still seen in agent 10.0.0.65 VM list 2025-10-31 19:38:44,658 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 2025-10-31 19:38:44,681 - WARNING - Pod uuid f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 is still seen in agent 10.0.0.65 VM list 2025-10-31 19:38:49,683 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 2025-10-31 19:38:49,690 - WARNING - Pod uuid f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 is still seen in agent 10.0.0.65 VM list 2025-10-31 19:38:54,692 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 2025-10-31 19:38:54,698 - WARNING - Pod uuid f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 is still seen in agent 10.0.0.65 VM list 2025-10-31 19:38:59,699 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 2025-10-31 19:38:59,705 - DEBUG - Pod f55fc1b6-cdc2-4fdd-952a-53f0126b81d0 is not in agent 10.0.0.65 VM list 2025-10-31 19:38:59,705 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:21876bfc-b691-11f0-8cce-fa163e1b5142 2025-10-31 19:38:59,710 - DEBUG - VMI 21876bfc-b691-11f0-8cce-fa163e1b5142 is removed from agent 10.0.0.65 2025-10-31 19:38:59,711 - INFO - Verified that pod ctest-busybox-pod-96756663 is removed in agent 2025-10-31 19:38:59,711 - INFO - Deleting Ingress : ctest-nginx-ingress-49473261 2025-10-31 19:38:59,729 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:38:59,733 - ERROR - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 still found in kube manager 2025-10-31 19:39:00,734 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:39:00,741 - ERROR - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 still found in kube manager 2025-10-31 19:39:01,741 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:39:01,745 - ERROR - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 still found in kube manager 2025-10-31 19:39:02,746 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:39:02,752 - ERROR - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 still found in kube manager 2025-10-31 19:39:03,752 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:39:03,757 - ERROR - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 still found in kube manager 2025-10-31 19:39:04,758 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:39:04,764 - ERROR - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 still found in kube manager 2025-10-31 19:39:05,766 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:39:05,771 - ERROR - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 still found in kube manager 2025-10-31 19:39:06,772 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:39:06,776 - ERROR - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 still found in kube manager 2025-10-31 19:39:07,778 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:39:07,783 - ERROR - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 still found in kube manager 2025-10-31 19:39:08,784 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=c49f7218-0e20-4c83-8097-af9982d88730 2025-10-31 19:39:08,788 - INFO - Ingress ctest-nginx-ingress-49473261 with uuid c49f7218-0e20-4c83-8097-af9982d88730 deleted successfully from kube manager 2025-10-31 19:39:08,789 - INFO - Verifications on Ingress %s cleanup passed 2025-10-31 19:39:08,790 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-10-31 19:39:08,900 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-31 19:39:08,909 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-10-31 19:39:08,909 - INFO - Deleting VN __public__ 2025-10-31 19:39:09,037 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/c699f5b3-3982-4a03-9172-ef77333a1ae7 2025-10-31 19:39:09,041 - DEBUG - Response Code: 404 2025-10-31 19:39:09,041 - DEBUG - Requesting: http://10.0.0.65:8082/domains 2025-10-31 19:39:09,045 - DEBUG - Requesting: http://10.0.0.65:8082/domain/252dd1eb-b22f-44c7-a25c-bbe9a34ef8f6 2025-10-31 19:39:09,073 - DEBUG - Requesting: http://10.0.0.65:8082/project/7c8b4bb8-79bc-439d-b45e-126cb331f18b 2025-10-31 19:39:09,138 - INFO - Validated that VN __public__ is not found in API Server 2025-10-31 19:39:09,138 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-10-31 19:39:09,146 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-31 19:39:09,151 - DEBUG - VN __public__ is not present in Agent 10.0.0.65 2025-10-31 19:39:09,151 - INFO - Validated that VN __public__ is not in any agent 2025-10-31 19:39:09,151 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-31 19:39:09,165 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-31 19:39:09,173 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-31 19:39:09,181 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-10-31 19:39:09,181 - INFO - Deleting pod default:ctest-nginx-pod-61377039 2025-10-31 19:39:09,196 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=79ba52f3-6a9d-41cf-abe2-1cc874c024e7 2025-10-31 19:39:09,211 - WARNING - Pod uuid 79ba52f3-6a9d-41cf-abe2-1cc874c024e7 is still seen in agent 10.0.0.65 VM list 2025-10-31 19:39:14,211 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=79ba52f3-6a9d-41cf-abe2-1cc874c024e7 2025-10-31 19:39:14,218 - DEBUG - Pod 79ba52f3-6a9d-41cf-abe2-1cc874c024e7 is not in agent 10.0.0.65 VM list 2025-10-31 19:39:14,218 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:16140adc-b691-11f0-8cce-fa163e1b5142 2025-10-31 19:39:14,223 - DEBUG - VMI 16140adc-b691-11f0-8cce-fa163e1b5142 is removed from agent 10.0.0.65 2025-10-31 19:39:14,224 - INFO - Verified that pod ctest-nginx-pod-61377039 is removed in agent 2025-10-31 19:39:14,224 - INFO - Deleting pod default:ctest-nginx-pod-33311041 2025-10-31 19:39:14,255 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=c1e52dea-af8b-4daf-a426-cd6f23bcafdf 2025-10-31 19:39:14,264 - WARNING - Pod uuid c1e52dea-af8b-4daf-a426-cd6f23bcafdf is still seen in agent 10.0.0.65 VM list 2025-10-31 19:39:19,265 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=c1e52dea-af8b-4daf-a426-cd6f23bcafdf 2025-10-31 19:39:19,270 - DEBUG - Pod c1e52dea-af8b-4daf-a426-cd6f23bcafdf is not in agent 10.0.0.65 VM list 2025-10-31 19:39:19,270 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:13fe8c40-b691-11f0-8cce-fa163e1b5142 2025-10-31 19:39:19,275 - DEBUG - VMI 13fe8c40-b691-11f0-8cce-fa163e1b5142 is removed from agent 10.0.0.65 2025-10-31 19:39:19,275 - INFO - Verified that pod ctest-nginx-pod-33311041 is removed in agent 2025-10-31 19:39:19,276 - INFO - Deleting service : ctest-nginx-svc-82495844 2025-10-31 19:39:19,821 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-31 19:39:19,828 - DEBUG - No XMPP flaps were noticed during the test