2026-08-25 14:06:39,400 - DEBUG - [10.0.0.161]: Running cmd : hostname
2026-08-25 14:06:39,840 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-2087-1
2026-08-25 14:06:39,840 - DEBUG - [10.0.0.161]: Running cmd : hostname -f
2026-08-25 14:06:39,896 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-2087-1
2026-08-25 14:06:39,897 - DEBUG - [10.0.0.161]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2026-08-25 14:06:40,054 - DEBUG - Output : NAMES
contrail_test_5k02gQ8g3
contrail-kubernetes-master-kubemanager-1
webui-job-1
webui-web-1
control-named-1
control-nodemgr-1
control-dns-1
control-control-1
config_database-nodemgr-1
config_database-rabbitmq-1
config_database-cassandra-1
config_database-zookeeper-1
config_api-devicemgr-1
config_api-svcmonitor-1
config_api-schema-1
config_api-nodemgr-1
config_api-dnsmasq-1
config_api-api-1
analytics_database-nodemgr-1
analytics_database-query-engine-1
analytics_database-cassandra-1
redis-redis-1
analytics_snmp-snmp-collector-1
analytics_snmp-nodemgr-1
analytics_snmp-topology-1
analytics_alarm-kafka-1
analytics_alarm-alarm-gen-1
analytics_alarm-nodemgr-1
analytics-api-1
analytics-collector-1
analytics-nodemgr-1
2026-08-25 14:06:40,054 - DEBUG - [10.0.0.161]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2026-08-25 14:06:40,133 - DEBUG - Output : 127.0.0.1/8
10.0.0.161/24
252.161.0.1/8
10.231.105.1/24
172.17.0.1/16
2026-08-25 14:06:40,133 - DEBUG - [10.0.0.161]: Running cmd : getent hosts 10.0.0.161 | head -n 1 | awk '{print $2}'
2026-08-25 14:06:40,232 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-2087-1
2026-08-25 14:06:40,361 - DEBUG - [10.0.0.177]: Running cmd : hostname
2026-08-25 14:06:40,618 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-2087-1
2026-08-25 14:06:40,618 - DEBUG - [10.0.0.177]: Running cmd : hostname -f
2026-08-25 14:06:40,652 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-2087-1
2026-08-25 14:06:40,653 - DEBUG - [10.0.0.177]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2026-08-25 14:06:40,708 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2026-08-25 14:06:40,708 - DEBUG - [10.0.0.177]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2026-08-25 14:06:40,778 - DEBUG - Output : 10.0.0.177/24
2026-08-25 14:06:40,778 - DEBUG - [10.0.0.177]: Running cmd : getent hosts 10.0.0.177 | head -n 1 | awk '{print $2}'
2026-08-25 14:06:40,817 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-2087-1
2026-08-25 14:06:40,953 - INFO - Using existing project ['default-domain', 'k8s-default'](43fefb03-b16c-48e0-87d9-8216fbf310e1)
2026-08-25 14:06:41,099 - INFO - Using existing project ['default-domain', 'k8s-default'](43fefb03-b16c-48e0-87d9-8216fbf310e1)
2026-08-25 14:06:41,274 - INFO - ================================================================================
2026-08-25 14:06:41,276 - INFO - STARTING TEST : test_ingress_ip_assignment
2026-08-25 14:06:41,277 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2026-08-25 14:06:41,614 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_AgentXmppConnectionStatusReq?
2026-08-25 14:06:41,629 - DEBUG - Nothing to compare xmpp stats {'10.0.0.177': {'10.0.0.161': '0'}} with
2026-08-25 14:06:41,629 - INFO - Initial checks done. Running the testcase now
2026-08-25 14:06:41,630 - INFO -
2026-08-25 14:06:41,634 - INFO - Namespace default already exists
2026-08-25 14:06:41,635 - DEBUG - [10.0.0.161]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2026-08-25 14:06:42,021 - DEBUG - Output : cluster_project={}
2026-08-25 14:06:42,192 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2026-08-25 14:06:42,192 - DEBUG - Project uuid in Contrail is 43fefb03-b16c-48e0-87d9-8216fbf310e1
2026-08-25 14:06:42,193 - DEBUG - [10.0.0.161]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2026-08-25 14:06:42,750 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2026-08-25 14:06:42,750 - DEBUG - Requesting: http://10.0.0.161:8108/Snh_NamespaceDatabaseList?namespace_uuid=b62640a2-404a-458e-901c-798d59db9967
2026-08-25 14:06:42,758 - INFO - Namespace default with uuid b62640a2-404a-458e-901c-798d59db9967 found in kube manager
2026-08-25 14:06:42,758 - INFO - Namespace default verification passed
2026-08-25 14:06:42,808 - DEBUG - Service ctest-nginx-svc-33000282 not present
2026-08-25 14:06:42,809 - INFO - Creating service ctest-nginx-svc-33000282
2026-08-25 14:06:42,931 - INFO - Created Service ctest-nginx-svc-33000282
2026-08-25 14:06:42,947 - DEBUG - Pod ctest-nginx-pod-79424637 not present
2026-08-25 14:06:42,948 - INFO - Creating Pod ctest-nginx-pod-79424637
2026-08-25 14:06:43,145 - DEBUG - Node for Pod ctest-nginx-pod-79424637 not yet populated
2026-08-25 14:06:46,154 - DEBUG - Node for Pod ctest-nginx-pod-79424637 not yet populated
2026-08-25 14:06:49,166 - DEBUG - Pod : ctest-nginx-pod-79424637 UUID is 8d32c209-081c-40b2-84d5-1bceb716e303
2026-08-25 14:06:49,169 - DEBUG - Pod ctest-nginx-pod-94744089 not present
2026-08-25 14:06:49,169 - INFO - Creating Pod ctest-nginx-pod-94744089
2026-08-25 14:06:49,235 - DEBUG - Pod : ctest-nginx-pod-94744089 UUID is be43f807-2b6b-4f8d-8e22-56a61fde54fe
2026-08-25 14:06:49,567 - INFO - Created VN __public__, UUID :f3016c3e-1d4a-47f8-8222-8d009c804cab
2026-08-25 14:06:49,818 - DEBUG - Requesting: http://10.0.0.161:8082/domains
2026-08-25 14:06:49,828 - DEBUG - Requesting: http://10.0.0.161:8082/domain/963e98b0-d96f-4780-ac39-8f8f40c38e65
2026-08-25 14:06:49,847 - DEBUG - Requesting: http://10.0.0.161:8082/project/43fefb03-b16c-48e0-87d9-8216fbf310e1
2026-08-25 14:06:49,954 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-network/f3016c3e-1d4a-47f8-8222-8d009c804cab
2026-08-25 14:06:49,971 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-network/f3016c3e-1d4a-47f8-8222-8d009c804cab
2026-08-25 14:06:49,983 - DEBUG - Requesting: http://10.0.0.161:8082/routing-instance/197b984c-0a1d-4cc4-b998-f2d0d51f82ee
2026-08-25 14:06:49,991 - DEBUG - Requesting: http://10.0.0.161:8082/routing-instance/197b984c-0a1d-4cc4-b998-f2d0d51f82ee
2026-08-25 14:06:50,000 - DEBUG - Requesting: http://10.0.0.161:8082/route-target/e1a9258f-227b-47cf-8f9a-43a3d92d3e6e
2026-08-25 14:06:50,011 - DEBUG - Route Targets: ['target:64512:8000006']
2026-08-25 14:06:50,011 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-network/f3016c3e-1d4a-47f8-8222-8d009c804cab
2026-08-25 14:06:50,026 - DEBUG - Requesting: http://10.0.0.161:8082/routing-instance/197b984c-0a1d-4cc4-b998-f2d0d51f82ee
2026-08-25 14:06:50,047 - INFO - Verified VN network id 13 for VN f3016c3e-1d4a-47f8-8222-8d009c804cab
2026-08-25 14:06:50,048 - INFO - Verifications in API Server for VN __public__ passed
2026-08-25 14:06:50,048 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-network/f3016c3e-1d4a-47f8-8222-8d009c804cab
2026-08-25 14:06:50,073 - DEBUG - Requesting: http://10.0.0.161:8082/routing-instance/197b984c-0a1d-4cc4-b998-f2d0d51f82ee
2026-08-25 14:06:50,092 - DEBUG - Requesting: http://10.0.0.161:8082/routing-instance/197b984c-0a1d-4cc4-b998-f2d0d51f82ee
2026-08-25 14:06:50,101 - DEBUG - Requesting: http://10.0.0.161:8082/route-target/e1a9258f-227b-47cf-8f9a-43a3d92d3e6e
2026-08-25 14:06:50,130 - DEBUG - Requesting: http://10.0.0.161:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2026-08-25 14:06:50,143 - DEBUG - Control-node 10.0.0.161 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17510395840228378616', 'uuid-lslong': '9377212407903964331'}, 'enable': 'true', 'created': '2026-08-25T14:06:49', 'last-modified': '2026-08-25T14:06:49', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.421628'}
2026-08-25 14:06:50,143 - DEBUG - Requesting: http://10.0.0.161:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2026-08-25 14:06:50,188 - DEBUG - Route Targets: ['target:64512:8000006']
2026-08-25 14:06:50,189 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2026-08-25 14:06:50,189 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2026-08-25 14:06:50,189 - DEBUG - Requesting: http://10.0.0.161:8082/domains
2026-08-25 14:06:50,196 - DEBUG - Requesting: http://10.0.0.161:8082/domain/963e98b0-d96f-4780-ac39-8f8f40c38e65
2026-08-25 14:06:50,208 - DEBUG - Requesting: http://10.0.0.161:8082/project/43fefb03-b16c-48e0-87d9-8216fbf310e1
2026-08-25 14:06:50,373 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-network/f3016c3e-1d4a-47f8-8222-8d009c804cab
2026-08-25 14:06:50,424 - DEBUG - =>VN __public__ has no policy to be verified
2026-08-25 14:06:50,424 - DEBUG - Verifying the vn in opserver
2026-08-25 14:06:50,424 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.161
2026-08-25 14:06:50,424 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-networks
2026-08-25 14:06:50,430 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.161:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2026-08-25 14:06:50,430 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2026-08-25 14:06:50,431 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-08-25 14:06:50,443 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-08-25 14:06:50,450 - DEBUG - Do not have enough data to verify VN in agent
2026-08-25 14:06:50,450 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-08-25 14:06:50,459 - DEBUG - VRF ids for VN __public__: {}
2026-08-25 14:06:50,675 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2026-08-25 14:06:50,842 - DEBUG - Requesting: http://10.0.0.161:8082/domains
2026-08-25 14:06:50,857 - DEBUG - Requesting: http://10.0.0.161:8082/domain/963e98b0-d96f-4780-ac39-8f8f40c38e65
2026-08-25 14:06:50,893 - DEBUG - Requesting: http://10.0.0.161:8082/project/43fefb03-b16c-48e0-87d9-8216fbf310e1
2026-08-25 14:06:50,952 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-network/f3016c3e-1d4a-47f8-8222-8d009c804cab
2026-08-25 14:06:50,996 - DEBUG - Requesting: http://10.0.0.161:8082/floating-ip-pool/44035260-de89-4e29-b307-283acdb95b04
2026-08-25 14:06:51,022 - DEBUG - Requesting: http://10.0.0.161:8082/domains
2026-08-25 14:06:51,034 - DEBUG - Requesting: http://10.0.0.161:8082/domain/963e98b0-d96f-4780-ac39-8f8f40c38e65
2026-08-25 14:06:51,042 - DEBUG - Requesting: http://10.0.0.161:8082/project/43fefb03-b16c-48e0-87d9-8216fbf310e1
2026-08-25 14:06:51,120 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-network/f3016c3e-1d4a-47f8-8222-8d009c804cab
2026-08-25 14:06:51,135 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2026-08-25 14:06:51,135 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2026-08-25 14:06:51,154 - DEBUG - Requesting: http://10.0.0.161:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2026-08-25 14:06:51,176 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2026-08-25 14:06:51,176 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2026-08-25 14:06:51,194 - DEBUG - Ingress ctest-nginx-ingress-96382723 not present
2026-08-25 14:06:51,195 - INFO - Creating Ingress ctest-nginx-ingress-96382723
2026-08-25 14:06:51,207 - INFO - Created Ingress ctest-nginx-ingress-96382723
2026-08-25 14:06:51,218 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-96382723
2026-08-25 14:06:54,226 - DEBUG - For Ingress ctest-nginx-ingress-96382723, Cluster IP: 10.47.255.249, External IPs ['200.150.102.252']
2026-08-25 14:06:54,226 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-96382723
2026-08-25 14:06:54,226 - DEBUG - Requesting: http://10.0.0.161:8108/Snh_LoadbalancerDatabaseList?x=c6e40e20-2e6b-4c5c-bb02-6393e45dfd7f
2026-08-25 14:06:54,234 - INFO - Ingress ctest-nginx-ingress-96382723 with uuid c6e40e20-2e6b-4c5c-bb02-6393e45dfd7f found in kube manager
2026-08-25 14:06:54,234 - INFO - Ingress ctest-nginx-ingress-96382723 verification passed
2026-08-25 14:06:54,241 - DEBUG - Pod ctest-busybox-pod-77455619 not present
2026-08-25 14:06:54,242 - INFO - Creating Pod ctest-busybox-pod-77455619
2026-08-25 14:06:54,294 - DEBUG - Pod : ctest-busybox-pod-77455619 UUID is 4d46fc99-e257-4d7e-9f1f-fd880fa37d44
2026-08-25 14:06:54,322 - DEBUG - Pod ctest-nginx-pod-79424637 not in running state.Currently in Pending
2026-08-25 14:06:59,358 - DEBUG - Pod ctest-nginx-pod-79424637 not in running state.Currently in Pending
2026-08-25 14:07:04,369 - DEBUG - Pod ctest-nginx-pod-79424637 not in running state.Currently in Pending
2026-08-25 14:07:09,377 - INFO - Pod ctest-nginx-pod-79424637 is in running state.Got IP 10.47.255.251
2026-08-25 14:07:09,436 - DEBUG - Pod ctest-nginx-pod-79424637 has vmi 36cda03e-a08e-11f1-8668-fa163e01341d
2026-08-25 14:07:09,436 - INFO - Verified pod ctest-nginx-pod-79424637 in contrail-api
2026-08-25 14:07:09,436 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:07:09,447 - DEBUG - VMI 36cda03e-a08e-11f1-8668-fa163e01341d is active in agent 10.0.0.177
2026-08-25 14:07:09,448 - INFO - Verified Pod ctest-nginx-pod-79424637 in agent 10.0.0.177
2026-08-25 14:07:09,448 - DEBUG - Requesting: http://10.0.0.161:8108/Snh_VirtualMachineDatabaseList?x=8d32c209-081c-40b2-84d5-1bceb716e303
2026-08-25 14:07:09,456 - INFO - Pod ctest-nginx-pod-79424637 with uuid 8d32c209-081c-40b2-84d5-1bceb716e303 found in kube manager
2026-08-25 14:07:09,456 - INFO - Pod ctest-nginx-pod-79424637 verification passed
2026-08-25 14:07:09,532 - DEBUG - [Pod ctest-nginx-pod-79424637] Cmd: echo ctest-nginx-pod-79424637 > /usr/share/nginx/html/index.html, Output:
2026-08-25 14:07:09,540 - INFO - Pod ctest-nginx-pod-94744089 is in running state.Got IP 10.47.255.250
2026-08-25 14:07:09,558 - DEBUG - Pod ctest-nginx-pod-94744089 has vmi 37761be2-a08e-11f1-8668-fa163e01341d
2026-08-25 14:07:09,558 - INFO - Verified pod ctest-nginx-pod-94744089 in contrail-api
2026-08-25 14:07:09,559 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:07:09,570 - DEBUG - VMI 37761be2-a08e-11f1-8668-fa163e01341d is active in agent 10.0.0.177
2026-08-25 14:07:09,570 - INFO - Verified Pod ctest-nginx-pod-94744089 in agent 10.0.0.177
2026-08-25 14:07:09,570 - DEBUG - Requesting: http://10.0.0.161:8108/Snh_VirtualMachineDatabaseList?x=be43f807-2b6b-4f8d-8e22-56a61fde54fe
2026-08-25 14:07:09,575 - INFO - Pod ctest-nginx-pod-94744089 with uuid be43f807-2b6b-4f8d-8e22-56a61fde54fe found in kube manager
2026-08-25 14:07:09,575 - INFO - Pod ctest-nginx-pod-94744089 verification passed
2026-08-25 14:07:09,636 - DEBUG - [Pod ctest-nginx-pod-94744089] Cmd: echo ctest-nginx-pod-94744089 > /usr/share/nginx/html/index.html, Output:
2026-08-25 14:07:09,645 - INFO - Pod ctest-busybox-pod-77455619 is in running state.Got IP 10.47.255.248
2026-08-25 14:07:09,665 - DEBUG - Pod ctest-busybox-pod-77455619 has vmi 3b05c8de-a08e-11f1-8668-fa163e01341d
2026-08-25 14:07:09,665 - INFO - Verified pod ctest-busybox-pod-77455619 in contrail-api
2026-08-25 14:07:09,666 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:07:09,678 - DEBUG - VMI 3b05c8de-a08e-11f1-8668-fa163e01341d is active in agent 10.0.0.177
2026-08-25 14:07:09,679 - INFO - Verified Pod ctest-busybox-pod-77455619 in agent 10.0.0.177
2026-08-25 14:07:09,679 - DEBUG - Requesting: http://10.0.0.161:8108/Snh_VirtualMachineDatabaseList?x=4d46fc99-e257-4d7e-9f1f-fd880fa37d44
2026-08-25 14:07:09,684 - INFO - Pod ctest-busybox-pod-77455619 with uuid 4d46fc99-e257-4d7e-9f1f-fd880fa37d44 found in kube manager
2026-08-25 14:07:09,684 - INFO - Pod ctest-busybox-pod-77455619 verification passed
2026-08-25 14:07:09,744 - DEBUG - [Pod ctest-busybox-pod-77455619] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-79424637
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2026-08-25 14:07:09,745 - DEBUG - [Pod Pod ctest-busybox-pod-77455619] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2026-08-25 14:07:09,745 - DEBUG - [Pod Pod ctest-busybox-pod-77455619] Cmd output: ctest-nginx-pod-79424637
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2026-08-25 14:07:09,808 - DEBUG - [Pod ctest-busybox-pod-77455619] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-94744089
2026-08-25 14:07:09,809 - DEBUG - [Pod Pod ctest-busybox-pod-77455619] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2026-08-25 14:07:09,809 - DEBUG - [Pod Pod ctest-busybox-pod-77455619] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-94744089
2026-08-25 14:07:09,809 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-79424637': 1, 'ctest-nginx-pod-94744089': 1}
2026-08-25 14:07:09,809 - WARNING - Nothing to delete parallely
2026-08-25 14:07:09,810 - INFO - Deleting pod default:ctest-busybox-pod-77455619
2026-08-25 14:07:09,868 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4d46fc99-e257-4d7e-9f1f-fd880fa37d44
2026-08-25 14:07:09,876 - WARNING - Pod uuid 4d46fc99-e257-4d7e-9f1f-fd880fa37d44 is still seen in agent 10.0.0.177 VM list
2026-08-25 14:07:14,878 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4d46fc99-e257-4d7e-9f1f-fd880fa37d44
2026-08-25 14:07:14,886 - WARNING - Pod uuid 4d46fc99-e257-4d7e-9f1f-fd880fa37d44 is still seen in agent 10.0.0.177 VM list
2026-08-25 14:07:19,886 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4d46fc99-e257-4d7e-9f1f-fd880fa37d44
2026-08-25 14:07:19,894 - WARNING - Pod uuid 4d46fc99-e257-4d7e-9f1f-fd880fa37d44 is still seen in agent 10.0.0.177 VM list
2026-08-25 14:07:24,895 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4d46fc99-e257-4d7e-9f1f-fd880fa37d44
2026-08-25 14:07:24,904 - WARNING - Pod uuid 4d46fc99-e257-4d7e-9f1f-fd880fa37d44 is still seen in agent 10.0.0.177 VM list
2026-08-25 14:07:29,904 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4d46fc99-e257-4d7e-9f1f-fd880fa37d44
2026-08-25 14:07:29,914 - WARNING - Pod uuid 4d46fc99-e257-4d7e-9f1f-fd880fa37d44 is still seen in agent 10.0.0.177 VM list
2026-08-25 14:07:34,915 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4d46fc99-e257-4d7e-9f1f-fd880fa37d44
2026-08-25 14:07:34,931 - WARNING - Pod uuid 4d46fc99-e257-4d7e-9f1f-fd880fa37d44 is still seen in agent 10.0.0.177 VM list
2026-08-25 14:07:39,932 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4d46fc99-e257-4d7e-9f1f-fd880fa37d44
2026-08-25 14:07:39,942 - WARNING - Pod uuid 4d46fc99-e257-4d7e-9f1f-fd880fa37d44 is still seen in agent 10.0.0.177 VM list
2026-08-25 14:07:44,943 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4d46fc99-e257-4d7e-9f1f-fd880fa37d44
2026-08-25 14:07:44,951 - DEBUG - Pod 4d46fc99-e257-4d7e-9f1f-fd880fa37d44 is not in agent 10.0.0.177 VM list
2026-08-25 14:07:44,951 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3b05c8de-a08e-11f1-8668-fa163e01341d
2026-08-25 14:07:44,959 - DEBUG - VMI 3b05c8de-a08e-11f1-8668-fa163e01341d is removed from agent 10.0.0.177
2026-08-25 14:07:44,959 - INFO - Verified that pod ctest-busybox-pod-77455619 is removed in agent
2026-08-25 14:07:44,959 - INFO - Deleting Ingress : ctest-nginx-ingress-96382723
2026-08-25 14:07:44,967 - DEBUG - Requesting: http://10.0.0.161:8108/Snh_LoadbalancerDatabaseList?x=c6e40e20-2e6b-4c5c-bb02-6393e45dfd7f
2026-08-25 14:07:44,972 - ERROR - Ingress ctest-nginx-ingress-96382723 with uuid c6e40e20-2e6b-4c5c-bb02-6393e45dfd7f still found in kube manager
2026-08-25 14:07:45,973 - DEBUG - Requesting: http://10.0.0.161:8108/Snh_LoadbalancerDatabaseList?x=c6e40e20-2e6b-4c5c-bb02-6393e45dfd7f
2026-08-25 14:07:45,978 - ERROR - Ingress ctest-nginx-ingress-96382723 with uuid c6e40e20-2e6b-4c5c-bb02-6393e45dfd7f still found in kube manager
2026-08-25 14:07:46,979 - DEBUG - Requesting: http://10.0.0.161:8108/Snh_LoadbalancerDatabaseList?x=c6e40e20-2e6b-4c5c-bb02-6393e45dfd7f
2026-08-25 14:07:46,984 - INFO - Ingress ctest-nginx-ingress-96382723 with uuid c6e40e20-2e6b-4c5c-bb02-6393e45dfd7f deleted successfully from kube manager
2026-08-25 14:07:46,984 - INFO - Verifications on Ingress %s cleanup passed
2026-08-25 14:07:46,984 - INFO - Deleting the FIP pool __fip_pool_public__
2026-08-25 14:07:47,107 - DEBUG - Requesting: http://10.0.0.161:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2026-08-25 14:07:47,120 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2026-08-25 14:07:47,120 - INFO - Deleting VN __public__
2026-08-25 14:07:47,262 - DEBUG - Requesting: http://10.0.0.161:8082/routing-instance/197b984c-0a1d-4cc4-b998-f2d0d51f82ee
2026-08-25 14:07:47,267 - DEBUG - Response Code: 404
2026-08-25 14:07:47,268 - DEBUG - Requesting: http://10.0.0.161:8082/domains
2026-08-25 14:07:47,273 - DEBUG - Requesting: http://10.0.0.161:8082/domain/963e98b0-d96f-4780-ac39-8f8f40c38e65
2026-08-25 14:07:47,293 - DEBUG - Requesting: http://10.0.0.161:8082/project/43fefb03-b16c-48e0-87d9-8216fbf310e1
2026-08-25 14:07:47,368 - INFO - Validated that VN __public__ is not found in API Server
2026-08-25 14:07:47,369 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2026-08-25 14:07:47,376 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-08-25 14:07:47,384 - DEBUG - VN __public__ is not present in Agent 10.0.0.177
2026-08-25 14:07:47,384 - INFO - Validated that VN __public__ is not in any agent
2026-08-25 14:07:47,384 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-08-25 14:07:47,395 - DEBUG - Requesting: http://10.0.0.161:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2026-08-25 14:07:47,404 - DEBUG - Requesting: http://10.0.0.161:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2026-08-25 14:07:47,410 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2026-08-25 14:07:47,411 - INFO - Deleting pod default:ctest-nginx-pod-94744089
2026-08-25 14:07:47,444 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=be43f807-2b6b-4f8d-8e22-56a61fde54fe
2026-08-25 14:07:47,453 - WARNING - Pod uuid be43f807-2b6b-4f8d-8e22-56a61fde54fe is still seen in agent 10.0.0.177 VM list
2026-08-25 14:07:52,454 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=be43f807-2b6b-4f8d-8e22-56a61fde54fe
2026-08-25 14:07:52,464 - DEBUG - Pod be43f807-2b6b-4f8d-8e22-56a61fde54fe is not in agent 10.0.0.177 VM list
2026-08-25 14:07:52,464 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:37761be2-a08e-11f1-8668-fa163e01341d
2026-08-25 14:07:52,473 - DEBUG - VMI 37761be2-a08e-11f1-8668-fa163e01341d is removed from agent 10.0.0.177
2026-08-25 14:07:52,473 - INFO - Verified that pod ctest-nginx-pod-94744089 is removed in agent
2026-08-25 14:07:52,474 - INFO - Deleting pod default:ctest-nginx-pod-79424637
2026-08-25 14:07:52,489 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=8d32c209-081c-40b2-84d5-1bceb716e303
2026-08-25 14:07:52,500 - WARNING - Pod uuid 8d32c209-081c-40b2-84d5-1bceb716e303 is still seen in agent 10.0.0.177 VM list
2026-08-25 14:07:57,501 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=8d32c209-081c-40b2-84d5-1bceb716e303
2026-08-25 14:07:57,508 - DEBUG - Pod 8d32c209-081c-40b2-84d5-1bceb716e303 is not in agent 10.0.0.177 VM list
2026-08-25 14:07:57,508 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:36cda03e-a08e-11f1-8668-fa163e01341d
2026-08-25 14:07:57,514 - DEBUG - VMI 36cda03e-a08e-11f1-8668-fa163e01341d is removed from agent 10.0.0.177
2026-08-25 14:07:57,514 - INFO - Verified that pod ctest-nginx-pod-79424637 is removed in agent
2026-08-25 14:07:57,514 - INFO - Deleting service : ctest-nginx-svc-33000282
2026-08-25 14:07:57,786 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_AgentXmppConnectionStatusReq?
2026-08-25 14:07:57,795 - DEBUG - No XMPP flaps were noticed during the test
2026-08-25 14:07:57,795 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:16]
2026-08-25 14:07:57,795 - INFO - --------------------------------------------------------------------------------
2026-08-25 14:08:04,435 - DEBUG - [10.0.0.161]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2026-08-25 14:08:04,585 - DEBUG - Output : cluster_project={}
2026-08-25 14:08:04,737 - DEBUG - [10.0.0.161]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2026-08-25 14:08:04,987 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2026-08-25 14:08:04,994 - DEBUG - [10.0.0.161]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2026-08-25 14:08:05,184 - DEBUG - Output : cluster_project={}
2026-08-25 14:08:15,438 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:08:15,593 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:08:20,716 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:08:20,764 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:08:25,872 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:08:25,915 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:08:26,005 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:08:26,092 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-08-25 14:08:26,390 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_AgentXmppConnectionStatusReq?
2026-08-25 14:08:29,093 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_ApplicationPolicySetReq?
2026-08-25 14:08:29,101 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_AclReq?
2026-08-25 14:08:29,131 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_AclReq?
2026-08-25 14:08:31,161 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_AclReq?
2026-08-25 14:11:31,697 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_AgentXmppConnectionStatusReq?
2026-08-25 14:11:31,755 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4119264e-de9a-4495-839d-79c0f18a5906
2026-08-25 14:11:31,769 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=5df04763-2d34-43d7-a53f-d7cb4abb1abb
2026-08-25 14:11:31,776 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=909df396-31f9-412d-b592-77c0d8bdee76
2026-08-25 14:11:31,781 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=fbbe7364-2d22-4f67-98b0-e101ae99917f
2026-08-25 14:11:31,788 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=60e9d4dd-36cd-451e-8232-1f8ccacdfce7
2026-08-25 14:11:31,794 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=a4cc84b5-d4c4-44ea-8ff1-0ef665fae29b
2026-08-25 14:11:31,802 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=8f60c2b0-85d1-48ae-ae32-1f0d6e1234df
2026-08-25 14:11:31,808 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=7f177b54-4f1f-4d04-ab9d-7e5e699b8772
2026-08-25 14:11:36,885 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=5df04763-2d34-43d7-a53f-d7cb4abb1abb
2026-08-25 14:11:36,887 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4119264e-de9a-4495-839d-79c0f18a5906
2026-08-25 14:11:36,889 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=60e9d4dd-36cd-451e-8232-1f8ccacdfce7
2026-08-25 14:11:36,890 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=fbbe7364-2d22-4f67-98b0-e101ae99917f
2026-08-25 14:11:36,891 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=909df396-31f9-412d-b592-77c0d8bdee76
2026-08-25 14:11:36,910 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6635d7d8-a08e-11f1-8668-fa163e01341d
2026-08-25 14:11:36,912 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=8f60c2b0-85d1-48ae-ae32-1f0d6e1234df
2026-08-25 14:11:36,913 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=a4cc84b5-d4c4-44ea-8ff1-0ef665fae29b
2026-08-25 14:11:36,915 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=7f177b54-4f1f-4d04-ab9d-7e5e699b8772
2026-08-25 14:11:36,932 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:656998bc-a08e-11f1-8668-fa163e01341d
2026-08-25 14:11:36,933 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:64d4965e-a08e-11f1-8668-fa163e01341d
2026-08-25 14:11:36,937 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:66d355bc-a08e-11f1-8668-fa163e01341d
2026-08-25 14:11:41,908 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=5df04763-2d34-43d7-a53f-d7cb4abb1abb
2026-08-25 14:11:41,911 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=fbbe7364-2d22-4f67-98b0-e101ae99917f
2026-08-25 14:11:41,912 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4119264e-de9a-4495-839d-79c0f18a5906
2026-08-25 14:11:41,914 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=909df396-31f9-412d-b592-77c0d8bdee76
2026-08-25 14:11:46,918 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=5df04763-2d34-43d7-a53f-d7cb4abb1abb
2026-08-25 14:11:46,929 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=909df396-31f9-412d-b592-77c0d8bdee76
2026-08-25 14:11:46,930 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4119264e-de9a-4495-839d-79c0f18a5906
2026-08-25 14:11:46,932 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=fbbe7364-2d22-4f67-98b0-e101ae99917f
2026-08-25 14:11:51,927 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=5df04763-2d34-43d7-a53f-d7cb4abb1abb
2026-08-25 14:11:51,942 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=909df396-31f9-412d-b592-77c0d8bdee76
2026-08-25 14:11:51,944 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4119264e-de9a-4495-839d-79c0f18a5906
2026-08-25 14:11:51,945 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=fbbe7364-2d22-4f67-98b0-e101ae99917f
2026-08-25 14:11:56,936 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=5df04763-2d34-43d7-a53f-d7cb4abb1abb
2026-08-25 14:11:56,960 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4119264e-de9a-4495-839d-79c0f18a5906
2026-08-25 14:11:56,962 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=fbbe7364-2d22-4f67-98b0-e101ae99917f
2026-08-25 14:11:56,964 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=909df396-31f9-412d-b592-77c0d8bdee76
2026-08-25 14:12:01,944 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=5df04763-2d34-43d7-a53f-d7cb4abb1abb
2026-08-25 14:12:01,975 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4119264e-de9a-4495-839d-79c0f18a5906
2026-08-25 14:12:01,978 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=909df396-31f9-412d-b592-77c0d8bdee76
2026-08-25 14:12:01,980 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=fbbe7364-2d22-4f67-98b0-e101ae99917f
2026-08-25 14:12:06,966 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=5df04763-2d34-43d7-a53f-d7cb4abb1abb
2026-08-25 14:12:06,974 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:68b30684-a08e-11f1-8668-fa163e01341d
2026-08-25 14:12:06,993 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=4119264e-de9a-4495-839d-79c0f18a5906
2026-08-25 14:12:06,995 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=909df396-31f9-412d-b592-77c0d8bdee76
2026-08-25 14:12:06,997 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_VmListReq?uuid=fbbe7364-2d22-4f67-98b0-e101ae99917f
2026-08-25 14:12:07,008 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:681fefde-a08e-11f1-8668-fa163e01341d
2026-08-25 14:12:07,010 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6787c42a-a08e-11f1-8668-fa163e01341d
2026-08-25 14:12:07,011 - DEBUG - Requesting: http://10.0.0.177:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:693e4140-a08e-11f1-8668-fa163e01341d