2026-02-20 09:15:06,854 - DEBUG - [10.0.0.50]: Running cmd : hostname 2026-02-20 09:15:07,167 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1907-1 2026-02-20 09:15:07,167 - DEBUG - [10.0.0.50]: Running cmd : hostname -f 2026-02-20 09:15:07,200 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1907-1 2026-02-20 09:15:07,201 - DEBUG - [10.0.0.50]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-20 09:15:07,298 - DEBUG - Output : NAMES contrail_test_xEcLCHx7X contrail-kubernetes-master-kubemanager-1 analytics_database-query-engine-1 analytics_database-cassandra-1 analytics_database-nodemgr-1 analytics_snmp-nodemgr-1 analytics_snmp-snmp-collector-1 analytics_snmp-topology-1 analytics_alarm-kafka-1 analytics_alarm-nodemgr-1 analytics_alarm-alarm-gen-1 analytics-api-1 analytics-collector-1 analytics-nodemgr-1 redis-redis-1 webui-web-1 webui-job-1 control-named-1 control-dns-1 control-control-1 control-nodemgr-1 config_database-nodemgr-1 config_database-rabbitmq-1 config_database-cassandra-1 config_database-zookeeper-1 config_api-devicemgr-1 config_api-svcmonitor-1 config_api-schema-1 config_api-api-1 config_api-dnsmasq-1 config_api-nodemgr-1 2026-02-20 09:15:07,299 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-02-20 09:15:07,332 - DEBUG - Output : 127.0.0.1/8 10.0.0.50/24 252.50.0.1/8 10.239.98.1/24 172.17.0.1/16 2026-02-20 09:15:07,332 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.0.0.50 | head -n 1 | awk '{print $2}' 2026-02-20 09:15:07,383 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1907-1 2026-02-20 09:15:07,473 - DEBUG - [10.0.0.129]: Running cmd : hostname 2026-02-20 09:15:07,708 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1907-1 2026-02-20 09:15:07,708 - DEBUG - [10.0.0.129]: Running cmd : hostname -f 2026-02-20 09:15:07,742 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1907-1 2026-02-20 09:15:07,742 - DEBUG - [10.0.0.129]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-20 09:15:07,818 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2026-02-20 09:15:07,819 - DEBUG - [10.0.0.129]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-02-20 09:15:07,853 - DEBUG - Output : 10.0.0.129/24 2026-02-20 09:15:07,853 - DEBUG - [10.0.0.129]: Running cmd : getent hosts 10.0.0.129 | head -n 1 | awk '{print $2}' 2026-02-20 09:15:07,887 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1907-1 2026-02-20 09:15:07,950 - INFO - Using existing project ['default-domain', 'k8s-default'](501f8325-0c18-427a-9907-e055a1ff0fa9) 2026-02-20 09:15:08,070 - INFO - Using existing project ['default-domain', 'k8s-default'](501f8325-0c18-427a-9907-e055a1ff0fa9) 2026-02-20 09:15:08,178 - INFO - ================================================================================ 2026-02-20 09:15:08,178 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-02-20 09:15:08,178 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-02-20 09:15:08,391 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-20 09:15:08,399 - DEBUG - Nothing to compare xmpp stats {'10.0.0.129': {'10.0.0.50': '0'}} with 2026-02-20 09:15:08,399 - INFO - Initial checks done. Running the testcase now 2026-02-20 09:15:08,400 - INFO - 2026-02-20 09:15:08,403 - INFO - Namespace default already exists 2026-02-20 09:15:08,403 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-20 09:15:08,537 - DEBUG - Output : cluster_project={} 2026-02-20 09:15:08,588 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-02-20 09:15:08,588 - DEBUG - Project uuid in Contrail is 501f8325-0c18-427a-9907-e055a1ff0fa9 2026-02-20 09:15:08,588 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-02-20 09:15:08,736 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-02-20 09:15:08,736 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_NamespaceDatabaseList?namespace_uuid=21b6113e-03e7-4c3f-9723-a2ffe531a7e5 2026-02-20 09:15:08,740 - INFO - Namespace default with uuid 21b6113e-03e7-4c3f-9723-a2ffe531a7e5 found in kube manager 2026-02-20 09:15:08,740 - INFO - Namespace default verification passed 2026-02-20 09:15:08,743 - DEBUG - Service ctest-nginx-svc-51791646 not present 2026-02-20 09:15:08,743 - INFO - Creating service ctest-nginx-svc-51791646 2026-02-20 09:15:08,763 - INFO - Created Service ctest-nginx-svc-51791646 2026-02-20 09:15:08,771 - DEBUG - Pod ctest-nginx-pod-99235534 not present 2026-02-20 09:15:08,778 - INFO - Creating Pod ctest-nginx-pod-99235534 2026-02-20 09:15:08,827 - DEBUG - Pod : ctest-nginx-pod-99235534 UUID is f3bfb842-03c2-45ca-b216-b316d8214ed1 2026-02-20 09:15:08,836 - DEBUG - Pod ctest-nginx-pod-80876478 not present 2026-02-20 09:15:08,837 - INFO - Creating Pod ctest-nginx-pod-80876478 2026-02-20 09:15:08,880 - DEBUG - Pod : ctest-nginx-pod-80876478 UUID is c61a1410-e895-4dc5-a1ad-6d932975f8e1 2026-02-20 09:15:09,035 - INFO - Created VN __public__, UUID :b2c4ab8d-4523-4350-a8c3-0e98885fb4a4 2026-02-20 09:15:09,278 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-02-20 09:15:09,282 - DEBUG - Requesting: http://10.0.0.50:8082/domain/30f6b841-1309-40f3-9c41-bedb694c0e97 2026-02-20 09:15:09,321 - DEBUG - Requesting: http://10.0.0.50:8082/project/501f8325-0c18-427a-9907-e055a1ff0fa9 2026-02-20 09:15:09,385 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/b2c4ab8d-4523-4350-a8c3-0e98885fb4a4 2026-02-20 09:15:09,394 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/b2c4ab8d-4523-4350-a8c3-0e98885fb4a4 2026-02-20 09:15:09,403 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a816a83d-7ffc-4a5f-a7eb-5bc7d67077ab 2026-02-20 09:15:09,410 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a816a83d-7ffc-4a5f-a7eb-5bc7d67077ab 2026-02-20 09:15:09,415 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/29f1796a-76c2-4b9b-af67-92f842919e13 2026-02-20 09:15:09,422 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-20 09:15:09,422 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/b2c4ab8d-4523-4350-a8c3-0e98885fb4a4 2026-02-20 09:15:09,430 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a816a83d-7ffc-4a5f-a7eb-5bc7d67077ab 2026-02-20 09:15:09,442 - INFO - Verified VN network id 13 for VN b2c4ab8d-4523-4350-a8c3-0e98885fb4a4 2026-02-20 09:15:09,443 - INFO - Verifications in API Server for VN __public__ passed 2026-02-20 09:15:09,443 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/b2c4ab8d-4523-4350-a8c3-0e98885fb4a4 2026-02-20 09:15:09,450 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a816a83d-7ffc-4a5f-a7eb-5bc7d67077ab 2026-02-20 09:15:09,472 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a816a83d-7ffc-4a5f-a7eb-5bc7d67077ab 2026-02-20 09:15:09,482 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/29f1796a-76c2-4b9b-af67-92f842919e13 2026-02-20 09:15:09,488 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-20 09:15:09,494 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12881609457425138512', 'uuid-lslong': '12160579467116262564'}, 'enable': 'true', 'created': '2026-02-20T09:15:08', 'last-modified': '2026-02-20T09:15:09', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.313902'} 2026-02-20 09:15:09,494 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 09:15:09,503 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-20 09:15:09,504 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-02-20 09:15:09,504 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-02-20 09:15:09,505 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-02-20 09:15:09,517 - DEBUG - Requesting: http://10.0.0.50:8082/domain/30f6b841-1309-40f3-9c41-bedb694c0e97 2026-02-20 09:15:09,562 - DEBUG - Requesting: http://10.0.0.50:8082/project/501f8325-0c18-427a-9907-e055a1ff0fa9 2026-02-20 09:15:09,624 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/b2c4ab8d-4523-4350-a8c3-0e98885fb4a4 2026-02-20 09:15:09,663 - DEBUG - =>VN __public__ has no policy to be verified 2026-02-20 09:15:09,663 - DEBUG - Verifying the vn in opserver 2026-02-20 09:15:09,663 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.50 2026-02-20 09:15:09,663 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-02-20 09:15:09,667 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-02-20 09:15:09,667 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-02-20 09:15:09,667 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 09:15:09,675 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 09:15:09,680 - DEBUG - Do not have enough data to verify VN in agent 2026-02-20 09:15:09,680 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 09:15:09,684 - DEBUG - VRF ids for VN __public__: {} 2026-02-20 09:15:09,799 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-02-20 09:15:09,909 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-02-20 09:15:09,914 - DEBUG - Requesting: http://10.0.0.50:8082/domain/30f6b841-1309-40f3-9c41-bedb694c0e97 2026-02-20 09:15:09,922 - DEBUG - Requesting: http://10.0.0.50:8082/project/501f8325-0c18-427a-9907-e055a1ff0fa9 2026-02-20 09:15:09,993 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/b2c4ab8d-4523-4350-a8c3-0e98885fb4a4 2026-02-20 09:15:10,005 - DEBUG - Requesting: http://10.0.0.50:8082/floating-ip-pool/ca5acebb-d4c8-4197-8207-10b5ce7fa3c7 2026-02-20 09:15:10,024 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-02-20 09:15:10,028 - DEBUG - Requesting: http://10.0.0.50:8082/domain/30f6b841-1309-40f3-9c41-bedb694c0e97 2026-02-20 09:15:10,038 - DEBUG - Requesting: http://10.0.0.50:8082/project/501f8325-0c18-427a-9907-e055a1ff0fa9 2026-02-20 09:15:10,136 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/b2c4ab8d-4523-4350-a8c3-0e98885fb4a4 2026-02-20 09:15:10,143 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-02-20 09:15:10,143 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-02-20 09:15:10,148 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-20 09:15:10,155 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-02-20 09:15:10,155 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-02-20 09:15:10,158 - DEBUG - Ingress ctest-nginx-ingress-96658397 not present 2026-02-20 09:15:10,159 - INFO - Creating Ingress ctest-nginx-ingress-96658397 2026-02-20 09:15:10,166 - INFO - Created Ingress ctest-nginx-ingress-96658397 2026-02-20 09:15:10,169 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-96658397 2026-02-20 09:15:13,175 - DEBUG - For Ingress ctest-nginx-ingress-96658397, Cluster IP: 10.47.255.249, External IPs ['202.15.60.60'] 2026-02-20 09:15:13,175 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-96658397 2026-02-20 09:15:13,175 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=1d0ec947-3283-432b-b3f9-6cb33c8a5bdd 2026-02-20 09:15:13,180 - INFO - Ingress ctest-nginx-ingress-96658397 with uuid 1d0ec947-3283-432b-b3f9-6cb33c8a5bdd found in kube manager 2026-02-20 09:15:13,181 - INFO - Ingress ctest-nginx-ingress-96658397 verification passed 2026-02-20 09:15:13,183 - DEBUG - Pod ctest-busybox-pod-88968953 not present 2026-02-20 09:15:13,184 - INFO - Creating Pod ctest-busybox-pod-88968953 2026-02-20 09:15:13,218 - DEBUG - Pod : ctest-busybox-pod-88968953 UUID is 9d55bce2-8227-4c18-9c20-e16ed9ee647a 2026-02-20 09:15:13,227 - DEBUG - Pod ctest-nginx-pod-99235534 not in running state.Currently in Pending 2026-02-20 09:15:18,234 - DEBUG - Pod ctest-nginx-pod-99235534 not in running state.Currently in Pending 2026-02-20 09:15:23,241 - INFO - Pod ctest-nginx-pod-99235534 is in running state.Got IP 10.47.255.251 2026-02-20 09:15:23,252 - DEBUG - Pod ctest-nginx-pod-99235534 has vmi a80f44c6-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:15:23,252 - INFO - Verified pod ctest-nginx-pod-99235534 in contrail-api 2026-02-20 09:15:23,252 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:15:23,263 - DEBUG - VMI a80f44c6-0e3c-11f1-91cc-fa163ecb9273 is active in agent 10.0.0.129 2026-02-20 09:15:23,263 - INFO - Verified Pod ctest-nginx-pod-99235534 in agent 10.0.0.129 2026-02-20 09:15:23,263 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=f3bfb842-03c2-45ca-b216-b316d8214ed1 2026-02-20 09:15:23,267 - INFO - Pod ctest-nginx-pod-99235534 with uuid f3bfb842-03c2-45ca-b216-b316d8214ed1 found in kube manager 2026-02-20 09:15:23,267 - INFO - Pod ctest-nginx-pod-99235534 verification passed 2026-02-20 09:15:23,329 - DEBUG - [Pod ctest-nginx-pod-99235534] Cmd: echo ctest-nginx-pod-99235534 > /usr/share/nginx/html/index.html, Output: 2026-02-20 09:15:23,336 - INFO - Pod ctest-nginx-pod-80876478 is in running state.Got IP 10.47.255.250 2026-02-20 09:15:23,348 - DEBUG - Pod ctest-nginx-pod-80876478 has vmi a87a397a-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:15:23,348 - INFO - Verified pod ctest-nginx-pod-80876478 in contrail-api 2026-02-20 09:15:23,348 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:15:23,356 - DEBUG - VMI a87a397a-0e3c-11f1-91cc-fa163ecb9273 is active in agent 10.0.0.129 2026-02-20 09:15:23,356 - INFO - Verified Pod ctest-nginx-pod-80876478 in agent 10.0.0.129 2026-02-20 09:15:23,356 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=c61a1410-e895-4dc5-a1ad-6d932975f8e1 2026-02-20 09:15:23,359 - INFO - Pod ctest-nginx-pod-80876478 with uuid c61a1410-e895-4dc5-a1ad-6d932975f8e1 found in kube manager 2026-02-20 09:15:23,360 - INFO - Pod ctest-nginx-pod-80876478 verification passed 2026-02-20 09:15:23,408 - DEBUG - [Pod ctest-nginx-pod-80876478] Cmd: echo ctest-nginx-pod-80876478 > /usr/share/nginx/html/index.html, Output: 2026-02-20 09:15:23,416 - DEBUG - Pod ctest-busybox-pod-88968953 not in running state.Currently in Pending 2026-02-20 09:15:28,425 - INFO - Pod ctest-busybox-pod-88968953 is in running state.Got IP 10.47.255.248 2026-02-20 09:15:28,481 - DEBUG - Pod ctest-busybox-pod-88968953 has vmi aa92562a-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:15:28,482 - INFO - Verified pod ctest-busybox-pod-88968953 in contrail-api 2026-02-20 09:15:28,482 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:15:28,493 - DEBUG - VMI aa92562a-0e3c-11f1-91cc-fa163ecb9273 is active in agent 10.0.0.129 2026-02-20 09:15:28,493 - INFO - Verified Pod ctest-busybox-pod-88968953 in agent 10.0.0.129 2026-02-20 09:15:28,493 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=9d55bce2-8227-4c18-9c20-e16ed9ee647a 2026-02-20 09:15:28,497 - INFO - Pod ctest-busybox-pod-88968953 with uuid 9d55bce2-8227-4c18-9c20-e16ed9ee647a found in kube manager 2026-02-20 09:15:28,497 - INFO - Pod ctest-busybox-pod-88968953 verification passed 2026-02-20 09:15:28,552 - DEBUG - [Pod ctest-busybox-pod-88968953] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) ctest-nginx-pod-80876478 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-20 09:15:28,552 - DEBUG - [Pod Pod ctest-busybox-pod-88968953] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-02-20 09:15:28,552 - DEBUG - [Pod Pod ctest-busybox-pod-88968953] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) ctest-nginx-pod-80876478 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-20 09:15:28,607 - DEBUG - [Pod ctest-busybox-pod-88968953] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-99235534 2026-02-20 09:15:28,607 - DEBUG - [Pod Pod ctest-busybox-pod-88968953] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-02-20 09:15:28,607 - DEBUG - [Pod Pod ctest-busybox-pod-88968953] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-99235534 2026-02-20 09:15:28,607 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-99235534': 1, 'ctest-nginx-pod-80876478': 1} 2026-02-20 09:15:28,607 - WARNING - Nothing to delete parallely 2026-02-20 09:15:28,608 - INFO - Deleting pod default:ctest-busybox-pod-88968953 2026-02-20 09:15:28,624 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=9d55bce2-8227-4c18-9c20-e16ed9ee647a 2026-02-20 09:15:28,630 - WARNING - Pod uuid 9d55bce2-8227-4c18-9c20-e16ed9ee647a is still seen in agent 10.0.0.129 VM list 2026-02-20 09:15:33,631 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=9d55bce2-8227-4c18-9c20-e16ed9ee647a 2026-02-20 09:15:33,636 - WARNING - Pod uuid 9d55bce2-8227-4c18-9c20-e16ed9ee647a is still seen in agent 10.0.0.129 VM list 2026-02-20 09:15:38,637 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=9d55bce2-8227-4c18-9c20-e16ed9ee647a 2026-02-20 09:15:38,643 - WARNING - Pod uuid 9d55bce2-8227-4c18-9c20-e16ed9ee647a is still seen in agent 10.0.0.129 VM list 2026-02-20 09:15:43,644 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=9d55bce2-8227-4c18-9c20-e16ed9ee647a 2026-02-20 09:15:43,649 - WARNING - Pod uuid 9d55bce2-8227-4c18-9c20-e16ed9ee647a is still seen in agent 10.0.0.129 VM list 2026-02-20 09:15:48,651 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=9d55bce2-8227-4c18-9c20-e16ed9ee647a 2026-02-20 09:15:48,656 - WARNING - Pod uuid 9d55bce2-8227-4c18-9c20-e16ed9ee647a is still seen in agent 10.0.0.129 VM list 2026-02-20 09:15:53,657 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=9d55bce2-8227-4c18-9c20-e16ed9ee647a 2026-02-20 09:15:53,663 - WARNING - Pod uuid 9d55bce2-8227-4c18-9c20-e16ed9ee647a is still seen in agent 10.0.0.129 VM list 2026-02-20 09:15:58,664 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=9d55bce2-8227-4c18-9c20-e16ed9ee647a 2026-02-20 09:15:58,670 - WARNING - Pod uuid 9d55bce2-8227-4c18-9c20-e16ed9ee647a is still seen in agent 10.0.0.129 VM list 2026-02-20 09:16:03,671 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=9d55bce2-8227-4c18-9c20-e16ed9ee647a 2026-02-20 09:16:03,677 - DEBUG - Pod 9d55bce2-8227-4c18-9c20-e16ed9ee647a is not in agent 10.0.0.129 VM list 2026-02-20 09:16:03,677 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:aa92562a-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:16:03,682 - DEBUG - VMI aa92562a-0e3c-11f1-91cc-fa163ecb9273 is removed from agent 10.0.0.129 2026-02-20 09:16:03,682 - INFO - Verified that pod ctest-busybox-pod-88968953 is removed in agent 2026-02-20 09:16:03,682 - INFO - Deleting Ingress : ctest-nginx-ingress-96658397 2026-02-20 09:16:03,688 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=1d0ec947-3283-432b-b3f9-6cb33c8a5bdd 2026-02-20 09:16:03,691 - ERROR - Ingress ctest-nginx-ingress-96658397 with uuid 1d0ec947-3283-432b-b3f9-6cb33c8a5bdd still found in kube manager 2026-02-20 09:16:04,692 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=1d0ec947-3283-432b-b3f9-6cb33c8a5bdd 2026-02-20 09:16:04,697 - INFO - Ingress ctest-nginx-ingress-96658397 with uuid 1d0ec947-3283-432b-b3f9-6cb33c8a5bdd deleted successfully from kube manager 2026-02-20 09:16:04,697 - INFO - Verifications on Ingress %s cleanup passed 2026-02-20 09:16:04,698 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-02-20 09:16:04,818 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-20 09:16:04,826 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-02-20 09:16:04,826 - INFO - Deleting VN __public__ 2026-02-20 09:16:04,989 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a816a83d-7ffc-4a5f-a7eb-5bc7d67077ab 2026-02-20 09:16:04,999 - DEBUG - Response Code: 404 2026-02-20 09:16:05,000 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-02-20 09:16:05,008 - DEBUG - Requesting: http://10.0.0.50:8082/domain/30f6b841-1309-40f3-9c41-bedb694c0e97 2026-02-20 09:16:05,015 - DEBUG - Requesting: http://10.0.0.50:8082/project/501f8325-0c18-427a-9907-e055a1ff0fa9 2026-02-20 09:16:05,046 - INFO - Validated that VN __public__ is not found in API Server 2026-02-20 09:16:05,046 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-02-20 09:16:05,051 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 09:16:05,067 - DEBUG - VN __public__ is not present in Agent 10.0.0.129 2026-02-20 09:16:05,067 - INFO - Validated that VN __public__ is not in any agent 2026-02-20 09:16:05,068 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 09:16:05,074 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 09:16:05,088 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-20 09:16:05,094 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-02-20 09:16:05,095 - INFO - Deleting pod default:ctest-nginx-pod-80876478 2026-02-20 09:16:05,126 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=c61a1410-e895-4dc5-a1ad-6d932975f8e1 2026-02-20 09:16:05,133 - WARNING - Pod uuid c61a1410-e895-4dc5-a1ad-6d932975f8e1 is still seen in agent 10.0.0.129 VM list 2026-02-20 09:16:10,135 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=c61a1410-e895-4dc5-a1ad-6d932975f8e1 2026-02-20 09:16:10,158 - DEBUG - Pod c61a1410-e895-4dc5-a1ad-6d932975f8e1 is not in agent 10.0.0.129 VM list 2026-02-20 09:16:10,158 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a87a397a-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:16:10,162 - DEBUG - VMI a87a397a-0e3c-11f1-91cc-fa163ecb9273 is removed from agent 10.0.0.129 2026-02-20 09:16:10,162 - INFO - Verified that pod ctest-nginx-pod-80876478 is removed in agent 2026-02-20 09:16:10,162 - INFO - Deleting pod default:ctest-nginx-pod-99235534 2026-02-20 09:16:10,175 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=f3bfb842-03c2-45ca-b216-b316d8214ed1 2026-02-20 09:16:10,192 - WARNING - Pod uuid f3bfb842-03c2-45ca-b216-b316d8214ed1 is still seen in agent 10.0.0.129 VM list 2026-02-20 09:16:15,193 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=f3bfb842-03c2-45ca-b216-b316d8214ed1 2026-02-20 09:16:15,198 - DEBUG - Pod f3bfb842-03c2-45ca-b216-b316d8214ed1 is not in agent 10.0.0.129 VM list 2026-02-20 09:16:15,198 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a80f44c6-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:16:15,203 - DEBUG - VMI a80f44c6-0e3c-11f1-91cc-fa163ecb9273 is removed from agent 10.0.0.129 2026-02-20 09:16:15,203 - INFO - Verified that pod ctest-nginx-pod-99235534 is removed in agent 2026-02-20 09:16:15,203 - INFO - Deleting service : ctest-nginx-svc-51791646 2026-02-20 09:16:15,447 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-20 09:16:15,452 - DEBUG - No XMPP flaps were noticed during the test 2026-02-20 09:16:15,452 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:07] 2026-02-20 09:16:15,452 - INFO - -------------------------------------------------------------------------------- 2026-02-20 09:16:21,879 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-20 09:16:21,997 - DEBUG - Output : cluster_project={} 2026-02-20 09:16:22,108 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-02-20 09:16:22,283 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-02-20 09:16:22,287 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-20 09:16:22,410 - DEBUG - Output : cluster_project={} 2026-02-20 09:16:37,550 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:16:37,589 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:16:37,664 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:16:37,738 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:16:37,770 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:16:37,848 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:16:37,890 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:16:37,965 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 09:16:38,206 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-20 09:16:39,919 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_ApplicationPolicySetReq? 2026-02-20 09:16:39,926 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_AclReq? 2026-02-20 09:16:39,955 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_AclReq? 2026-02-20 09:16:41,983 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_AclReq? 2026-02-20 09:19:48,541 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-20 09:19:48,604 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=ed947de6-e2b8-44c8-8bd4-047c1a53bfdc 2026-02-20 09:19:48,652 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=ea128661-902c-41db-9f18-f826e151f19d 2026-02-20 09:19:48,662 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=bd73dbe3-dab1-431f-a9ad-b6dc381cc592 2026-02-20 09:19:48,692 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=219132e3-0cfc-49de-9d3c-02ccb18db22b 2026-02-20 09:19:48,705 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=41c41193-7ad6-4000-97b9-024ec598c9c6 2026-02-20 09:19:48,720 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=21da39ba-4de3-42db-8dde-4428426d5a17 2026-02-20 09:19:48,728 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=491361b1-0e8c-4dcf-a945-caa30701f365 2026-02-20 09:19:48,735 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=bc673ea2-66ad-4f5f-bce5-99260e3664a8 2026-02-20 09:19:53,743 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=bd73dbe3-dab1-431f-a9ad-b6dc381cc592 2026-02-20 09:19:53,745 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=ea128661-902c-41db-9f18-f826e151f19d 2026-02-20 09:19:53,746 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=ed947de6-e2b8-44c8-8bd4-047c1a53bfdc 2026-02-20 09:19:53,756 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d355b6a6-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:19:53,773 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=41c41193-7ad6-4000-97b9-024ec598c9c6 2026-02-20 09:19:53,780 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d4c6d7c2-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:19:53,782 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=219132e3-0cfc-49de-9d3c-02ccb18db22b 2026-02-20 09:19:53,783 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=bc673ea2-66ad-4f5f-bce5-99260e3664a8 2026-02-20 09:19:53,785 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=491361b1-0e8c-4dcf-a945-caa30701f365 2026-02-20 09:19:53,786 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=21da39ba-4de3-42db-8dde-4428426d5a17 2026-02-20 09:19:53,798 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d45f1d44-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:19:53,799 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d3cc02ca-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:19:58,756 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=bd73dbe3-dab1-431f-a9ad-b6dc381cc592 2026-02-20 09:19:58,757 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=ea128661-902c-41db-9f18-f826e151f19d 2026-02-20 09:19:58,803 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=21da39ba-4de3-42db-8dde-4428426d5a17 2026-02-20 09:19:58,804 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=491361b1-0e8c-4dcf-a945-caa30701f365 2026-02-20 09:20:03,766 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=bd73dbe3-dab1-431f-a9ad-b6dc381cc592 2026-02-20 09:20:03,767 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=ea128661-902c-41db-9f18-f826e151f19d 2026-02-20 09:20:03,811 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=21da39ba-4de3-42db-8dde-4428426d5a17 2026-02-20 09:20:03,813 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=491361b1-0e8c-4dcf-a945-caa30701f365 2026-02-20 09:20:08,777 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=bd73dbe3-dab1-431f-a9ad-b6dc381cc592 2026-02-20 09:20:08,779 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=ea128661-902c-41db-9f18-f826e151f19d 2026-02-20 09:20:08,826 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=491361b1-0e8c-4dcf-a945-caa30701f365 2026-02-20 09:20:08,827 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=21da39ba-4de3-42db-8dde-4428426d5a17 2026-02-20 09:20:13,786 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=bd73dbe3-dab1-431f-a9ad-b6dc381cc592 2026-02-20 09:20:13,788 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=ea128661-902c-41db-9f18-f826e151f19d 2026-02-20 09:20:13,835 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=491361b1-0e8c-4dcf-a945-caa30701f365 2026-02-20 09:20:13,836 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=21da39ba-4de3-42db-8dde-4428426d5a17 2026-02-20 09:20:18,795 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=bd73dbe3-dab1-431f-a9ad-b6dc381cc592 2026-02-20 09:20:18,797 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=ea128661-902c-41db-9f18-f826e151f19d 2026-02-20 09:20:18,845 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=491361b1-0e8c-4dcf-a945-caa30701f365 2026-02-20 09:20:18,846 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=21da39ba-4de3-42db-8dde-4428426d5a17 2026-02-20 09:20:23,807 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=bd73dbe3-dab1-431f-a9ad-b6dc381cc592 2026-02-20 09:20:23,808 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=ea128661-902c-41db-9f18-f826e151f19d 2026-02-20 09:20:23,815 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d53d6ea0-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:20:23,817 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d5c9db10-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:20:23,855 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=21da39ba-4de3-42db-8dde-4428426d5a17 2026-02-20 09:20:23,856 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_VmListReq?uuid=491361b1-0e8c-4dcf-a945-caa30701f365 2026-02-20 09:20:23,865 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d628f9ba-0e3c-11f1-91cc-fa163ecb9273 2026-02-20 09:20:23,867 - DEBUG - Requesting: http://10.0.0.129:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d6953486-0e3c-11f1-91cc-fa163ecb9273