2026-02-11 18:22:57,494 - DEBUG - [10.0.0.32]: Running cmd : hostname 2026-02-11 18:22:57,839 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1884-1 2026-02-11 18:22:57,839 - DEBUG - [10.0.0.32]: Running cmd : hostname -f 2026-02-11 18:22:57,880 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1884-1 2026-02-11 18:22:57,881 - DEBUG - [10.0.0.32]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-11 18:22:57,983 - DEBUG - Output : NAMES contrail_test_yq6J54GSZ contrail-kubernetes-master-kubemanager-1 analytics_database-nodemgr-1 analytics_database-cassandra-1 analytics_database-query-engine-1 analytics_snmp-nodemgr-1 analytics_snmp-snmp-collector-1 analytics_snmp-topology-1 analytics_alarm-nodemgr-1 analytics_alarm-kafka-1 analytics_alarm-alarm-gen-1 analytics-api-1 analytics-collector-1 analytics-nodemgr-1 redis-redis-1 webui-job-1 webui-web-1 webui-node-init-1 control-named-1 control-nodemgr-1 control-control-1 control-dns-1 config_database-cassandra-1 config_database-nodemgr-1 config_database-rabbitmq-1 config_database-zookeeper-1 config_api-devicemgr-1 config_api-api-1 config_api-svcmonitor-1 config_api-dnsmasq-1 config_api-nodemgr-1 config_api-schema-1 2026-02-11 18:22:57,985 - DEBUG - [10.0.0.32]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-02-11 18:22:58,019 - DEBUG - Output : 127.0.0.1/8 10.0.0.32/24 252.32.0.1/8 10.35.199.1/24 172.17.0.1/16 2026-02-11 18:22:58,020 - DEBUG - [10.0.0.32]: Running cmd : getent hosts 10.0.0.32 | head -n 1 | awk '{print $2}' 2026-02-11 18:22:58,053 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1884-1 2026-02-11 18:22:58,163 - DEBUG - [10.0.0.41]: Running cmd : hostname 2026-02-11 18:22:58,388 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1884-1 2026-02-11 18:22:58,388 - DEBUG - [10.0.0.41]: Running cmd : hostname -f 2026-02-11 18:22:58,422 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1884-1 2026-02-11 18:22:58,423 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-11 18:22:58,499 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2026-02-11 18:22:58,499 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-02-11 18:22:58,524 - DEBUG - Output : 10.0.0.41/24 2026-02-11 18:22:58,524 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.0.0.41 | head -n 1 | awk '{print $2}' 2026-02-11 18:22:58,559 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1884-1 2026-02-11 18:22:58,673 - INFO - Using existing project ['default-domain', 'k8s-default'](f555397a-fa1d-4c90-bf16-4b755e726cb4) 2026-02-11 18:22:58,800 - INFO - Using existing project ['default-domain', 'k8s-default'](f555397a-fa1d-4c90-bf16-4b755e726cb4) 2026-02-11 18:22:58,929 - INFO - ================================================================================ 2026-02-11 18:22:58,929 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-02-11 18:22:58,929 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-02-11 18:22:59,107 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-11 18:22:59,116 - DEBUG - Nothing to compare xmpp stats {'10.0.0.41': {'10.0.0.32': '0'}} with 2026-02-11 18:22:59,116 - INFO - Initial checks done. Running the testcase now 2026-02-11 18:22:59,116 - INFO - 2026-02-11 18:22:59,121 - INFO - Namespace default already exists 2026-02-11 18:22:59,122 - DEBUG - [10.0.0.32]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-11 18:22:59,269 - DEBUG - Output : cluster_project={} 2026-02-11 18:22:59,319 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-02-11 18:22:59,319 - DEBUG - Project uuid in Contrail is f555397a-fa1d-4c90-bf16-4b755e726cb4 2026-02-11 18:22:59,319 - DEBUG - [10.0.0.32]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-02-11 18:22:59,477 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-02-11 18:22:59,477 - DEBUG - Requesting: http://10.0.0.32:8108/Snh_NamespaceDatabaseList?namespace_uuid=d63b7ee2-8a44-4417-b93f-a258a50e3443 2026-02-11 18:22:59,481 - INFO - Namespace default with uuid d63b7ee2-8a44-4417-b93f-a258a50e3443 found in kube manager 2026-02-11 18:22:59,481 - INFO - Namespace default verification passed 2026-02-11 18:22:59,491 - DEBUG - Service ctest-nginx-svc-17101849 not present 2026-02-11 18:22:59,492 - INFO - Creating service ctest-nginx-svc-17101849 2026-02-11 18:22:59,646 - INFO - Created Service ctest-nginx-svc-17101849 2026-02-11 18:22:59,653 - DEBUG - Pod ctest-nginx-pod-56453939 not present 2026-02-11 18:22:59,653 - INFO - Creating Pod ctest-nginx-pod-56453939 2026-02-11 18:22:59,731 - DEBUG - Node for Pod ctest-nginx-pod-56453939 not yet populated 2026-02-11 18:23:02,739 - DEBUG - Pod : ctest-nginx-pod-56453939 UUID is 4fc22d94-888b-414f-8882-848d1a0262db 2026-02-11 18:23:02,741 - DEBUG - Pod ctest-nginx-pod-25736917 not present 2026-02-11 18:23:02,742 - INFO - Creating Pod ctest-nginx-pod-25736917 2026-02-11 18:23:02,775 - DEBUG - Pod : ctest-nginx-pod-25736917 UUID is b26ee687-0892-48a2-8c19-3f1d02c1285e 2026-02-11 18:23:02,982 - INFO - Created VN __public__, UUID :9ce59275-c7e0-4f92-a543-0c0aeada4fd7 2026-02-11 18:23:03,258 - DEBUG - Requesting: http://10.0.0.32:8082/domains 2026-02-11 18:23:03,267 - DEBUG - Requesting: http://10.0.0.32:8082/domain/9bec6a31-cfab-469d-809e-bfa9a4dff341 2026-02-11 18:23:03,290 - DEBUG - Requesting: http://10.0.0.32:8082/project/f555397a-fa1d-4c90-bf16-4b755e726cb4 2026-02-11 18:23:03,367 - DEBUG - Requesting: http://10.0.0.32:8082/virtual-network/9ce59275-c7e0-4f92-a543-0c0aeada4fd7 2026-02-11 18:23:03,393 - DEBUG - Requesting: http://10.0.0.32:8082/virtual-network/9ce59275-c7e0-4f92-a543-0c0aeada4fd7 2026-02-11 18:23:03,402 - DEBUG - Requesting: http://10.0.0.32:8082/routing-instance/0196d32b-af23-4024-b8a4-7b114b353415 2026-02-11 18:23:03,410 - DEBUG - Requesting: http://10.0.0.32:8082/routing-instance/0196d32b-af23-4024-b8a4-7b114b353415 2026-02-11 18:23:03,416 - DEBUG - Requesting: http://10.0.0.32:8082/route-target/3c5204b2-1877-4d5d-8de3-aefe7dc455ab 2026-02-11 18:23:03,422 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-11 18:23:03,422 - DEBUG - Requesting: http://10.0.0.32:8082/virtual-network/9ce59275-c7e0-4f92-a543-0c0aeada4fd7 2026-02-11 18:23:03,430 - DEBUG - Requesting: http://10.0.0.32:8082/routing-instance/0196d32b-af23-4024-b8a4-7b114b353415 2026-02-11 18:23:03,452 - INFO - Verified VN network id 13 for VN 9ce59275-c7e0-4f92-a543-0c0aeada4fd7 2026-02-11 18:23:03,452 - INFO - Verifications in API Server for VN __public__ passed 2026-02-11 18:23:03,452 - DEBUG - Requesting: http://10.0.0.32:8082/virtual-network/9ce59275-c7e0-4f92-a543-0c0aeada4fd7 2026-02-11 18:23:03,461 - DEBUG - Requesting: http://10.0.0.32:8082/routing-instance/0196d32b-af23-4024-b8a4-7b114b353415 2026-02-11 18:23:03,469 - DEBUG - Requesting: http://10.0.0.32:8082/routing-instance/0196d32b-af23-4024-b8a4-7b114b353415 2026-02-11 18:23:03,476 - DEBUG - Requesting: http://10.0.0.32:8082/route-target/3c5204b2-1877-4d5d-8de3-aefe7dc455ab 2026-02-11 18:23:03,482 - DEBUG - Requesting: http://10.0.0.32:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-11 18:23:03,500 - DEBUG - Control-node 10.0.0.32 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '11305603474145693586', 'uuid-lslong': '11908375080727105495'}, 'enable': 'true', 'created': '2026-02-11T18:23:02', 'last-modified': '2026-02-11T18:23:03', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.299445'} 2026-02-11 18:23:03,500 - DEBUG - Requesting: http://10.0.0.32:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-11 18:23:03,518 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-11 18:23:03,518 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-02-11 18:23:03,519 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-02-11 18:23:03,519 - DEBUG - Requesting: http://10.0.0.32:8082/domains 2026-02-11 18:23:03,525 - DEBUG - Requesting: http://10.0.0.32:8082/domain/9bec6a31-cfab-469d-809e-bfa9a4dff341 2026-02-11 18:23:03,534 - DEBUG - Requesting: http://10.0.0.32:8082/project/f555397a-fa1d-4c90-bf16-4b755e726cb4 2026-02-11 18:23:03,569 - DEBUG - Requesting: http://10.0.0.32:8082/virtual-network/9ce59275-c7e0-4f92-a543-0c0aeada4fd7 2026-02-11 18:23:03,582 - DEBUG - =>VN __public__ has no policy to be verified 2026-02-11 18:23:03,582 - DEBUG - Verifying the vn in opserver 2026-02-11 18:23:03,582 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.32 2026-02-11 18:23:03,583 - DEBUG - Requesting: http://10.0.0.32:8081/analytics/uves/virtual-networks 2026-02-11 18:23:03,587 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.32:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-02-11 18:23:03,587 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-02-11 18:23:03,588 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-11 18:23:03,597 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-11 18:23:03,603 - DEBUG - Do not have enough data to verify VN in agent 2026-02-11 18:23:03,604 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-11 18:23:03,613 - DEBUG - VRF ids for VN __public__: {} 2026-02-11 18:23:03,731 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-02-11 18:23:03,865 - DEBUG - Requesting: http://10.0.0.32:8082/domains 2026-02-11 18:23:03,872 - DEBUG - Requesting: http://10.0.0.32:8082/domain/9bec6a31-cfab-469d-809e-bfa9a4dff341 2026-02-11 18:23:03,880 - DEBUG - Requesting: http://10.0.0.32:8082/project/f555397a-fa1d-4c90-bf16-4b755e726cb4 2026-02-11 18:23:03,918 - DEBUG - Requesting: http://10.0.0.32:8082/virtual-network/9ce59275-c7e0-4f92-a543-0c0aeada4fd7 2026-02-11 18:23:03,928 - DEBUG - Requesting: http://10.0.0.32:8082/floating-ip-pool/44ea3a95-8494-4b0a-87c3-e631fa0dda3b 2026-02-11 18:23:03,934 - DEBUG - Requesting: http://10.0.0.32:8082/domains 2026-02-11 18:23:03,939 - DEBUG - Requesting: http://10.0.0.32:8082/domain/9bec6a31-cfab-469d-809e-bfa9a4dff341 2026-02-11 18:23:03,946 - DEBUG - Requesting: http://10.0.0.32:8082/project/f555397a-fa1d-4c90-bf16-4b755e726cb4 2026-02-11 18:23:03,991 - DEBUG - Requesting: http://10.0.0.32:8082/virtual-network/9ce59275-c7e0-4f92-a543-0c0aeada4fd7 2026-02-11 18:23:04,001 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-02-11 18:23:04,001 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-02-11 18:23:04,009 - DEBUG - Requesting: http://10.0.0.32:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-11 18:23:04,014 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-02-11 18:23:04,015 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-02-11 18:23:04,018 - DEBUG - Ingress ctest-nginx-ingress-55502686 not present 2026-02-11 18:23:04,019 - INFO - Creating Ingress ctest-nginx-ingress-55502686 2026-02-11 18:23:04,029 - INFO - Created Ingress ctest-nginx-ingress-55502686 2026-02-11 18:23:04,033 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-55502686 2026-02-11 18:23:07,040 - DEBUG - For Ingress ctest-nginx-ingress-55502686, Cluster IP: 10.47.255.249, External IPs ['86.102.49.188'] 2026-02-11 18:23:07,040 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-55502686 2026-02-11 18:23:07,040 - DEBUG - Requesting: http://10.0.0.32:8108/Snh_LoadbalancerDatabaseList?x=ea54fc7d-52bf-4cfa-a14f-858709d08d07 2026-02-11 18:23:07,048 - INFO - Ingress ctest-nginx-ingress-55502686 with uuid ea54fc7d-52bf-4cfa-a14f-858709d08d07 found in kube manager 2026-02-11 18:23:07,048 - INFO - Ingress ctest-nginx-ingress-55502686 verification passed 2026-02-11 18:23:07,051 - DEBUG - Pod ctest-busybox-pod-98988968 not present 2026-02-11 18:23:07,052 - INFO - Creating Pod ctest-busybox-pod-98988968 2026-02-11 18:23:07,102 - DEBUG - Pod : ctest-busybox-pod-98988968 UUID is 95c36de7-bec3-4042-bc16-277dde54d532 2026-02-11 18:23:07,110 - DEBUG - Pod ctest-nginx-pod-56453939 not in running state.Currently in Pending 2026-02-11 18:23:12,120 - DEBUG - Pod ctest-nginx-pod-56453939 not in running state.Currently in Pending 2026-02-11 18:23:17,127 - INFO - Pod ctest-nginx-pod-56453939 is in running state.Got IP 10.47.255.251 2026-02-11 18:23:17,185 - DEBUG - Pod ctest-nginx-pod-56453939 has vmi b2cde290-0776-11f1-937f-fa163eda8f87 2026-02-11 18:23:17,186 - INFO - Verified pod ctest-nginx-pod-56453939 in contrail-api 2026-02-11 18:23:17,186 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:23:17,205 - DEBUG - VMI b2cde290-0776-11f1-937f-fa163eda8f87 is active in agent 10.0.0.41 2026-02-11 18:23:17,206 - INFO - Verified Pod ctest-nginx-pod-56453939 in agent 10.0.0.41 2026-02-11 18:23:17,207 - DEBUG - Requesting: http://10.0.0.32:8108/Snh_VirtualMachineDatabaseList?x=4fc22d94-888b-414f-8882-848d1a0262db 2026-02-11 18:23:17,238 - INFO - Pod ctest-nginx-pod-56453939 with uuid 4fc22d94-888b-414f-8882-848d1a0262db found in kube manager 2026-02-11 18:23:17,238 - INFO - Pod ctest-nginx-pod-56453939 verification passed 2026-02-11 18:23:17,320 - DEBUG - [Pod ctest-nginx-pod-56453939] Cmd: echo ctest-nginx-pod-56453939 > /usr/share/nginx/html/index.html, Output: 2026-02-11 18:23:17,338 - INFO - Pod ctest-nginx-pod-25736917 is in running state.Got IP 10.47.255.250 2026-02-11 18:23:17,355 - DEBUG - Pod ctest-nginx-pod-25736917 has vmi b3d220d4-0776-11f1-937f-fa163eda8f87 2026-02-11 18:23:17,355 - INFO - Verified pod ctest-nginx-pod-25736917 in contrail-api 2026-02-11 18:23:17,355 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:23:17,364 - DEBUG - VMI b3d220d4-0776-11f1-937f-fa163eda8f87 is active in agent 10.0.0.41 2026-02-11 18:23:17,364 - INFO - Verified Pod ctest-nginx-pod-25736917 in agent 10.0.0.41 2026-02-11 18:23:17,364 - DEBUG - Requesting: http://10.0.0.32:8108/Snh_VirtualMachineDatabaseList?x=b26ee687-0892-48a2-8c19-3f1d02c1285e 2026-02-11 18:23:17,368 - INFO - Pod ctest-nginx-pod-25736917 with uuid b26ee687-0892-48a2-8c19-3f1d02c1285e found in kube manager 2026-02-11 18:23:17,368 - INFO - Pod ctest-nginx-pod-25736917 verification passed 2026-02-11 18:23:17,418 - DEBUG - [Pod ctest-nginx-pod-25736917] Cmd: echo ctest-nginx-pod-25736917 > /usr/share/nginx/html/index.html, Output: 2026-02-11 18:23:17,426 - DEBUG - Pod ctest-busybox-pod-98988968 not in running state.Currently in Pending 2026-02-11 18:23:22,435 - INFO - Pod ctest-busybox-pod-98988968 is in running state.Got IP 10.47.255.248 2026-02-11 18:23:22,449 - DEBUG - Pod ctest-busybox-pod-98988968 has vmi b65ec41a-0776-11f1-937f-fa163eda8f87 2026-02-11 18:23:22,449 - INFO - Verified pod ctest-busybox-pod-98988968 in contrail-api 2026-02-11 18:23:22,449 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:23:22,460 - DEBUG - VMI b65ec41a-0776-11f1-937f-fa163eda8f87 is active in agent 10.0.0.41 2026-02-11 18:23:22,460 - INFO - Verified Pod ctest-busybox-pod-98988968 in agent 10.0.0.41 2026-02-11 18:23:22,460 - DEBUG - Requesting: http://10.0.0.32:8108/Snh_VirtualMachineDatabaseList?x=95c36de7-bec3-4042-bc16-277dde54d532 2026-02-11 18:23:22,464 - INFO - Pod ctest-busybox-pod-98988968 with uuid 95c36de7-bec3-4042-bc16-277dde54d532 found in kube manager 2026-02-11 18:23:22,464 - INFO - Pod ctest-busybox-pod-98988968 verification passed 2026-02-11 18:23:22,534 - DEBUG - [Pod ctest-busybox-pod-98988968] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-56453939 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-11 18:23:22,534 - DEBUG - [Pod Pod ctest-busybox-pod-98988968] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-02-11 18:23:22,534 - DEBUG - [Pod Pod ctest-busybox-pod-98988968] Cmd output: ctest-nginx-pod-56453939 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-11 18:23:22,587 - DEBUG - [Pod ctest-busybox-pod-98988968] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-25736917 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-11 18:23:22,587 - DEBUG - [Pod Pod ctest-busybox-pod-98988968] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-02-11 18:23:22,587 - DEBUG - [Pod Pod ctest-busybox-pod-98988968] Cmd output: ctest-nginx-pod-25736917 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-11 18:23:22,587 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-56453939': 1, 'ctest-nginx-pod-25736917': 1} 2026-02-11 18:23:22,587 - WARNING - Nothing to delete parallely 2026-02-11 18:23:22,587 - INFO - Deleting pod default:ctest-busybox-pod-98988968 2026-02-11 18:23:22,600 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=95c36de7-bec3-4042-bc16-277dde54d532 2026-02-11 18:23:22,607 - WARNING - Pod uuid 95c36de7-bec3-4042-bc16-277dde54d532 is still seen in agent 10.0.0.41 VM list 2026-02-11 18:23:27,608 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=95c36de7-bec3-4042-bc16-277dde54d532 2026-02-11 18:23:27,615 - WARNING - Pod uuid 95c36de7-bec3-4042-bc16-277dde54d532 is still seen in agent 10.0.0.41 VM list 2026-02-11 18:23:32,617 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=95c36de7-bec3-4042-bc16-277dde54d532 2026-02-11 18:23:32,626 - WARNING - Pod uuid 95c36de7-bec3-4042-bc16-277dde54d532 is still seen in agent 10.0.0.41 VM list 2026-02-11 18:23:37,627 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=95c36de7-bec3-4042-bc16-277dde54d532 2026-02-11 18:23:37,633 - WARNING - Pod uuid 95c36de7-bec3-4042-bc16-277dde54d532 is still seen in agent 10.0.0.41 VM list 2026-02-11 18:23:42,634 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=95c36de7-bec3-4042-bc16-277dde54d532 2026-02-11 18:23:42,641 - WARNING - Pod uuid 95c36de7-bec3-4042-bc16-277dde54d532 is still seen in agent 10.0.0.41 VM list 2026-02-11 18:23:47,642 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=95c36de7-bec3-4042-bc16-277dde54d532 2026-02-11 18:23:47,653 - WARNING - Pod uuid 95c36de7-bec3-4042-bc16-277dde54d532 is still seen in agent 10.0.0.41 VM list 2026-02-11 18:23:52,654 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=95c36de7-bec3-4042-bc16-277dde54d532 2026-02-11 18:23:52,661 - WARNING - Pod uuid 95c36de7-bec3-4042-bc16-277dde54d532 is still seen in agent 10.0.0.41 VM list 2026-02-11 18:23:57,662 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=95c36de7-bec3-4042-bc16-277dde54d532 2026-02-11 18:23:57,669 - DEBUG - Pod 95c36de7-bec3-4042-bc16-277dde54d532 is not in agent 10.0.0.41 VM list 2026-02-11 18:23:57,669 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b65ec41a-0776-11f1-937f-fa163eda8f87 2026-02-11 18:23:57,675 - DEBUG - VMI b65ec41a-0776-11f1-937f-fa163eda8f87 is removed from agent 10.0.0.41 2026-02-11 18:23:57,675 - INFO - Verified that pod ctest-busybox-pod-98988968 is removed in agent 2026-02-11 18:23:57,675 - INFO - Deleting Ingress : ctest-nginx-ingress-55502686 2026-02-11 18:23:57,683 - DEBUG - Requesting: http://10.0.0.32:8108/Snh_LoadbalancerDatabaseList?x=ea54fc7d-52bf-4cfa-a14f-858709d08d07 2026-02-11 18:23:57,688 - ERROR - Ingress ctest-nginx-ingress-55502686 with uuid ea54fc7d-52bf-4cfa-a14f-858709d08d07 still found in kube manager 2026-02-11 18:23:58,689 - DEBUG - Requesting: http://10.0.0.32:8108/Snh_LoadbalancerDatabaseList?x=ea54fc7d-52bf-4cfa-a14f-858709d08d07 2026-02-11 18:23:58,695 - INFO - Ingress ctest-nginx-ingress-55502686 with uuid ea54fc7d-52bf-4cfa-a14f-858709d08d07 deleted successfully from kube manager 2026-02-11 18:23:58,697 - INFO - Verifications on Ingress %s cleanup passed 2026-02-11 18:23:58,697 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-02-11 18:23:58,833 - DEBUG - Requesting: http://10.0.0.32:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-11 18:23:58,842 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-02-11 18:23:58,842 - INFO - Deleting VN __public__ 2026-02-11 18:23:59,051 - DEBUG - Requesting: http://10.0.0.32:8082/routing-instance/0196d32b-af23-4024-b8a4-7b114b353415 2026-02-11 18:23:59,066 - DEBUG - Response Code: 404 2026-02-11 18:23:59,066 - DEBUG - Requesting: http://10.0.0.32:8082/domains 2026-02-11 18:23:59,072 - DEBUG - Requesting: http://10.0.0.32:8082/domain/9bec6a31-cfab-469d-809e-bfa9a4dff341 2026-02-11 18:23:59,079 - DEBUG - Requesting: http://10.0.0.32:8082/project/f555397a-fa1d-4c90-bf16-4b755e726cb4 2026-02-11 18:23:59,148 - INFO - Validated that VN __public__ is not found in API Server 2026-02-11 18:23:59,148 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-02-11 18:23:59,169 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-11 18:23:59,181 - DEBUG - VN __public__ is not present in Agent 10.0.0.41 2026-02-11 18:23:59,181 - INFO - Validated that VN __public__ is not in any agent 2026-02-11 18:23:59,182 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-11 18:23:59,188 - DEBUG - Requesting: http://10.0.0.32:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-11 18:23:59,199 - DEBUG - Requesting: http://10.0.0.32:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-11 18:23:59,208 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-02-11 18:23:59,209 - INFO - Deleting pod default:ctest-nginx-pod-25736917 2026-02-11 18:23:59,237 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=b26ee687-0892-48a2-8c19-3f1d02c1285e 2026-02-11 18:23:59,244 - WARNING - Pod uuid b26ee687-0892-48a2-8c19-3f1d02c1285e is still seen in agent 10.0.0.41 VM list 2026-02-11 18:24:04,245 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=b26ee687-0892-48a2-8c19-3f1d02c1285e 2026-02-11 18:24:04,252 - DEBUG - Pod b26ee687-0892-48a2-8c19-3f1d02c1285e is not in agent 10.0.0.41 VM list 2026-02-11 18:24:04,252 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b3d220d4-0776-11f1-937f-fa163eda8f87 2026-02-11 18:24:04,258 - DEBUG - VMI b3d220d4-0776-11f1-937f-fa163eda8f87 is removed from agent 10.0.0.41 2026-02-11 18:24:04,258 - INFO - Verified that pod ctest-nginx-pod-25736917 is removed in agent 2026-02-11 18:24:04,259 - INFO - Deleting pod default:ctest-nginx-pod-56453939 2026-02-11 18:24:04,279 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=4fc22d94-888b-414f-8882-848d1a0262db 2026-02-11 18:24:04,289 - WARNING - Pod uuid 4fc22d94-888b-414f-8882-848d1a0262db is still seen in agent 10.0.0.41 VM list 2026-02-11 18:24:09,290 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=4fc22d94-888b-414f-8882-848d1a0262db 2026-02-11 18:24:09,297 - DEBUG - Pod 4fc22d94-888b-414f-8882-848d1a0262db is not in agent 10.0.0.41 VM list 2026-02-11 18:24:09,297 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b2cde290-0776-11f1-937f-fa163eda8f87 2026-02-11 18:24:09,302 - DEBUG - VMI b2cde290-0776-11f1-937f-fa163eda8f87 is removed from agent 10.0.0.41 2026-02-11 18:24:09,302 - INFO - Verified that pod ctest-nginx-pod-56453939 is removed in agent 2026-02-11 18:24:09,303 - INFO - Deleting service : ctest-nginx-svc-17101849 2026-02-11 18:24:09,535 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-11 18:24:09,543 - DEBUG - No XMPP flaps were noticed during the test 2026-02-11 18:24:09,543 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:11] 2026-02-11 18:24:09,543 - INFO - -------------------------------------------------------------------------------- 2026-02-11 18:24:16,034 - DEBUG - [10.0.0.32]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-11 18:24:16,177 - DEBUG - Output : cluster_project={} 2026-02-11 18:24:16,295 - DEBUG - [10.0.0.32]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-02-11 18:24:16,435 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-02-11 18:24:16,440 - DEBUG - [10.0.0.32]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-11 18:24:16,564 - DEBUG - Output : cluster_project={} 2026-02-11 18:24:26,653 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:24:31,699 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:24:31,786 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:24:31,822 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:24:36,906 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:24:36,946 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:24:37,026 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:24:37,127 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-11 18:24:37,356 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-11 18:24:39,969 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_ApplicationPolicySetReq? 2026-02-11 18:24:39,975 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AclReq? 2026-02-11 18:24:40,000 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AclReq? 2026-02-11 18:24:42,034 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AclReq? 2026-02-11 18:26:50,346 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-11 18:26:50,391 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=8bed45c7-daf3-445c-8070-a8bafbc4b039 2026-02-11 18:26:50,396 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=4d20ce46-c5cf-4c69-a0c6-10bf88ee8858 2026-02-11 18:26:50,402 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ebdc3e0f-a981-41fd-ac9a-12f256c604d3 2026-02-11 18:26:50,408 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=178785b6-2f79-44ba-9201-38534a8e578a 2026-02-11 18:26:50,415 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=7cea1c8f-1b3f-448c-b858-8d2713febd46 2026-02-11 18:26:50,420 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=b5ed80d7-8ccb-493f-b785-c3ad3505773d 2026-02-11 18:26:50,425 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ab92b0f4-929e-494b-929e-756283cb4254 2026-02-11 18:26:50,431 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=14c0f66a-14b8-4781-aa99-75fec65e9dfb 2026-02-11 18:26:55,428 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=4d20ce46-c5cf-4c69-a0c6-10bf88ee8858 2026-02-11 18:26:55,429 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=8bed45c7-daf3-445c-8070-a8bafbc4b039 2026-02-11 18:26:55,434 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ebdc3e0f-a981-41fd-ac9a-12f256c604d3 2026-02-11 18:26:55,435 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=178785b6-2f79-44ba-9201-38534a8e578a 2026-02-11 18:26:55,438 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e15b125e-0776-11f1-937f-fa163eda8f87 2026-02-11 18:26:55,446 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e0b2eb24-0776-11f1-937f-fa163eda8f87 2026-02-11 18:26:55,449 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e032e9d8-0776-11f1-937f-fa163eda8f87 2026-02-11 18:26:55,450 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=b5ed80d7-8ccb-493f-b785-c3ad3505773d 2026-02-11 18:26:55,452 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=7cea1c8f-1b3f-448c-b858-8d2713febd46 2026-02-11 18:26:55,457 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ab92b0f4-929e-494b-929e-756283cb4254 2026-02-11 18:26:55,458 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=14c0f66a-14b8-4781-aa99-75fec65e9dfb 2026-02-11 18:26:55,470 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e1d57e86-0776-11f1-937f-fa163eda8f87 2026-02-11 18:27:00,439 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=4d20ce46-c5cf-4c69-a0c6-10bf88ee8858 2026-02-11 18:27:00,465 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=b5ed80d7-8ccb-493f-b785-c3ad3505773d 2026-02-11 18:27:00,466 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=7cea1c8f-1b3f-448c-b858-8d2713febd46 2026-02-11 18:27:00,470 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ab92b0f4-929e-494b-929e-756283cb4254 2026-02-11 18:27:05,446 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=4d20ce46-c5cf-4c69-a0c6-10bf88ee8858 2026-02-11 18:27:05,474 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=7cea1c8f-1b3f-448c-b858-8d2713febd46 2026-02-11 18:27:05,476 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=b5ed80d7-8ccb-493f-b785-c3ad3505773d 2026-02-11 18:27:05,477 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ab92b0f4-929e-494b-929e-756283cb4254 2026-02-11 18:27:10,454 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=4d20ce46-c5cf-4c69-a0c6-10bf88ee8858 2026-02-11 18:27:10,485 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=7cea1c8f-1b3f-448c-b858-8d2713febd46 2026-02-11 18:27:10,487 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ab92b0f4-929e-494b-929e-756283cb4254 2026-02-11 18:27:10,488 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=b5ed80d7-8ccb-493f-b785-c3ad3505773d 2026-02-11 18:27:15,462 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=4d20ce46-c5cf-4c69-a0c6-10bf88ee8858 2026-02-11 18:27:15,498 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=7cea1c8f-1b3f-448c-b858-8d2713febd46 2026-02-11 18:27:15,500 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ab92b0f4-929e-494b-929e-756283cb4254 2026-02-11 18:27:15,500 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=b5ed80d7-8ccb-493f-b785-c3ad3505773d 2026-02-11 18:27:20,470 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=4d20ce46-c5cf-4c69-a0c6-10bf88ee8858 2026-02-11 18:27:20,511 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ab92b0f4-929e-494b-929e-756283cb4254 2026-02-11 18:27:20,513 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=7cea1c8f-1b3f-448c-b858-8d2713febd46 2026-02-11 18:27:20,514 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=b5ed80d7-8ccb-493f-b785-c3ad3505773d 2026-02-11 18:27:25,484 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=4d20ce46-c5cf-4c69-a0c6-10bf88ee8858 2026-02-11 18:27:25,491 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e3351034-0776-11f1-937f-fa163eda8f87 2026-02-11 18:27:25,524 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=b5ed80d7-8ccb-493f-b785-c3ad3505773d 2026-02-11 18:27:25,526 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ab92b0f4-929e-494b-929e-756283cb4254 2026-02-11 18:27:25,527 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=7cea1c8f-1b3f-448c-b858-8d2713febd46 2026-02-11 18:27:25,538 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e3a5fcb8-0776-11f1-937f-fa163eda8f87 2026-02-11 18:27:25,539 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e251d3f0-0776-11f1-937f-fa163eda8f87 2026-02-11 18:27:25,541 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e2c46a32-0776-11f1-937f-fa163eda8f87