2026-03-05 01:43:41,008 - DEBUG - [10.0.0.27]: Running cmd : hostname
2026-03-05 01:43:41,251 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1921-1
2026-03-05 01:43:41,251 - DEBUG - [10.0.0.27]: Running cmd : hostname -f
2026-03-05 01:43:41,284 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1921-1
2026-03-05 01:43:41,285 - DEBUG - [10.0.0.27]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2026-03-05 01:43:41,385 - DEBUG - Output : NAMES
contrail_test_dTJN2Jswk
webui-job-1
webui-web-1
contrail-kubernetes-master-kubemanager-1
analytics_snmp-topology-1
analytics_snmp-snmp-collector-1
analytics_snmp-nodemgr-1
analytics_alarm-kafka-1
analytics_alarm-nodemgr-1
analytics_alarm-alarm-gen-1
analytics-collector-1
analytics-api-1
analytics-nodemgr-1
analytics_database-query-engine-1
analytics_database-nodemgr-1
analytics_database-cassandra-1
redis-redis-1
control-named-1
control-nodemgr-1
control-control-1
control-dns-1
config_database-rabbitmq-1
config_database-zookeeper-1
config_database-nodemgr-1
config_database-cassandra-1
config_api-devicemgr-1
config_api-api-1
config_api-schema-1
config_api-svcmonitor-1
config_api-dnsmasq-1
config_api-nodemgr-1
2026-03-05 01:43:41,385 - DEBUG - [10.0.0.27]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2026-03-05 01:43:41,418 - DEBUG - Output : 127.0.0.1/8
10.0.0.27/24
252.27.0.1/8
10.55.41.1/24
172.17.0.1/16
2026-03-05 01:43:41,419 - DEBUG - [10.0.0.27]: Running cmd : getent hosts 10.0.0.27 | head -n 1 | awk '{print $2}'
2026-03-05 01:43:41,452 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1921-1
2026-03-05 01:43:41,559 - DEBUG - [10.0.0.240]: Running cmd : hostname
2026-03-05 01:43:41,808 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1921-1
2026-03-05 01:43:41,808 - DEBUG - [10.0.0.240]: Running cmd : hostname -f
2026-03-05 01:43:41,877 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1921-1
2026-03-05 01:43:41,877 - DEBUG - [10.0.0.240]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2026-03-05 01:43:41,931 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2026-03-05 01:43:41,931 - DEBUG - [10.0.0.240]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2026-03-05 01:43:41,965 - DEBUG - Output : 10.0.0.240/24
2026-03-05 01:43:41,965 - DEBUG - [10.0.0.240]: Running cmd : getent hosts 10.0.0.240 | head -n 1 | awk '{print $2}'
2026-03-05 01:43:42,036 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1921-1
2026-03-05 01:43:42,102 - INFO - Using existing project ['default-domain', 'k8s-default'](1034e79d-df62-4019-9087-126393eb9448)
2026-03-05 01:43:42,214 - INFO - Using existing project ['default-domain', 'k8s-default'](1034e79d-df62-4019-9087-126393eb9448)
2026-03-05 01:43:42,323 - INFO - ================================================================================
2026-03-05 01:43:42,324 - INFO - STARTING TEST : test_ingress_ip_assignment
2026-03-05 01:43:42,324 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2026-03-05 01:43:42,566 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_AgentXmppConnectionStatusReq?
2026-03-05 01:43:42,572 - DEBUG - Nothing to compare xmpp stats {'10.0.0.240': {'10.0.0.27': '0'}} with
2026-03-05 01:43:42,572 - INFO - Initial checks done. Running the testcase now
2026-03-05 01:43:42,573 - INFO -
2026-03-05 01:43:42,577 - INFO - Namespace default already exists
2026-03-05 01:43:42,577 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2026-03-05 01:43:42,714 - DEBUG - Output : cluster_project={}
2026-03-05 01:43:42,752 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2026-03-05 01:43:42,752 - DEBUG - Project uuid in Contrail is 1034e79d-df62-4019-9087-126393eb9448
2026-03-05 01:43:42,752 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2026-03-05 01:43:42,869 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2026-03-05 01:43:42,869 - DEBUG - Requesting: http://10.0.0.27:8108/Snh_NamespaceDatabaseList?namespace_uuid=c17520d1-0072-4915-98c9-c8e837ae3f61
2026-03-05 01:43:42,873 - INFO - Namespace default with uuid c17520d1-0072-4915-98c9-c8e837ae3f61 found in kube manager
2026-03-05 01:43:42,873 - INFO - Namespace default verification passed
2026-03-05 01:43:42,876 - DEBUG - Service ctest-nginx-svc-58391939 not present
2026-03-05 01:43:42,876 - INFO - Creating service ctest-nginx-svc-58391939
2026-03-05 01:43:42,906 - INFO - Created Service ctest-nginx-svc-58391939
2026-03-05 01:43:42,915 - DEBUG - Pod ctest-nginx-pod-90335849 not present
2026-03-05 01:43:42,915 - INFO - Creating Pod ctest-nginx-pod-90335849
2026-03-05 01:43:42,954 - DEBUG - Node for Pod ctest-nginx-pod-90335849 not yet populated
2026-03-05 01:43:45,961 - DEBUG - Pod : ctest-nginx-pod-90335849 UUID is e63a95b0-06b0-41bb-be5d-d8d321e9451f
2026-03-05 01:43:45,963 - DEBUG - Pod ctest-nginx-pod-22246185 not present
2026-03-05 01:43:45,963 - INFO - Creating Pod ctest-nginx-pod-22246185
2026-03-05 01:43:45,992 - DEBUG - Pod : ctest-nginx-pod-22246185 UUID is 4d676906-879d-4feb-89d4-b1b4f792c22c
2026-03-05 01:43:46,188 - INFO - Created VN __public__, UUID :4866c538-a2ce-4c11-8a88-0d147d468bdc
2026-03-05 01:43:46,402 - DEBUG - Requesting: http://10.0.0.27:8082/domains
2026-03-05 01:43:46,406 - DEBUG - Requesting: http://10.0.0.27:8082/domain/5725e86a-1270-4266-9169-c72cce701556
2026-03-05 01:43:46,431 - DEBUG - Requesting: http://10.0.0.27:8082/project/1034e79d-df62-4019-9087-126393eb9448
2026-03-05 01:43:46,490 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4866c538-a2ce-4c11-8a88-0d147d468bdc
2026-03-05 01:43:46,513 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4866c538-a2ce-4c11-8a88-0d147d468bdc
2026-03-05 01:43:46,519 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1489ef8d-ec60-4c0a-a8f8-f5aebd9e008d
2026-03-05 01:43:46,524 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1489ef8d-ec60-4c0a-a8f8-f5aebd9e008d
2026-03-05 01:43:46,528 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/b2452422-6580-481f-ab5e-fb31059184dd
2026-03-05 01:43:46,532 - DEBUG - Route Targets: ['target:64512:8000006']
2026-03-05 01:43:46,532 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4866c538-a2ce-4c11-8a88-0d147d468bdc
2026-03-05 01:43:46,538 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1489ef8d-ec60-4c0a-a8f8-f5aebd9e008d
2026-03-05 01:43:46,547 - INFO - Verified VN network id 13 for VN 4866c538-a2ce-4c11-8a88-0d147d468bdc
2026-03-05 01:43:46,547 - INFO - Verifications in API Server for VN __public__ passed
2026-03-05 01:43:46,547 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4866c538-a2ce-4c11-8a88-0d147d468bdc
2026-03-05 01:43:46,553 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1489ef8d-ec60-4c0a-a8f8-f5aebd9e008d
2026-03-05 01:43:46,558 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1489ef8d-ec60-4c0a-a8f8-f5aebd9e008d
2026-03-05 01:43:46,566 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/b2452422-6580-481f-ab5e-fb31059184dd
2026-03-05 01:43:46,572 - DEBUG - Requesting: http://10.0.0.27:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2026-03-05 01:43:46,578 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5217074065395567633', 'uuid-lslong': '9982242955718986716'}, 'enable': 'true', 'created': '2026-03-05T01:43:46', 'last-modified': '2026-03-05T01:43:46', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.235931'}
2026-03-05 01:43:46,578 - DEBUG - Requesting: http://10.0.0.27:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2026-03-05 01:43:46,594 - DEBUG - Route Targets: ['target:64512:8000006']
2026-03-05 01:43:46,594 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2026-03-05 01:43:46,595 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2026-03-05 01:43:46,595 - DEBUG - Requesting: http://10.0.0.27:8082/domains
2026-03-05 01:43:46,604 - DEBUG - Requesting: http://10.0.0.27:8082/domain/5725e86a-1270-4266-9169-c72cce701556
2026-03-05 01:43:46,610 - DEBUG - Requesting: http://10.0.0.27:8082/project/1034e79d-df62-4019-9087-126393eb9448
2026-03-05 01:43:46,636 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4866c538-a2ce-4c11-8a88-0d147d468bdc
2026-03-05 01:43:46,653 - DEBUG - =>VN __public__ has no policy to be verified
2026-03-05 01:43:46,653 - DEBUG - Verifying the vn in opserver
2026-03-05 01:43:46,653 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.27
2026-03-05 01:43:46,653 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks
2026-03-05 01:43:46,659 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2026-03-05 01:43:46,659 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2026-03-05 01:43:46,659 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-03-05 01:43:46,669 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-03-05 01:43:46,673 - DEBUG - Do not have enough data to verify VN in agent
2026-03-05 01:43:46,673 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-03-05 01:43:46,678 - DEBUG - VRF ids for VN __public__: {}
2026-03-05 01:43:46,762 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2026-03-05 01:43:46,846 - DEBUG - Requesting: http://10.0.0.27:8082/domains
2026-03-05 01:43:46,856 - DEBUG - Requesting: http://10.0.0.27:8082/domain/5725e86a-1270-4266-9169-c72cce701556
2026-03-05 01:43:46,862 - DEBUG - Requesting: http://10.0.0.27:8082/project/1034e79d-df62-4019-9087-126393eb9448
2026-03-05 01:43:46,901 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4866c538-a2ce-4c11-8a88-0d147d468bdc
2026-03-05 01:43:46,909 - DEBUG - Requesting: http://10.0.0.27:8082/floating-ip-pool/5c3d48cd-9fda-4d64-8d7d-e6a792965200
2026-03-05 01:43:46,914 - DEBUG - Requesting: http://10.0.0.27:8082/domains
2026-03-05 01:43:46,917 - DEBUG - Requesting: http://10.0.0.27:8082/domain/5725e86a-1270-4266-9169-c72cce701556
2026-03-05 01:43:46,922 - DEBUG - Requesting: http://10.0.0.27:8082/project/1034e79d-df62-4019-9087-126393eb9448
2026-03-05 01:43:46,958 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/4866c538-a2ce-4c11-8a88-0d147d468bdc
2026-03-05 01:43:46,965 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2026-03-05 01:43:46,965 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2026-03-05 01:43:46,970 - DEBUG - Requesting: http://10.0.0.27:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2026-03-05 01:43:46,975 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2026-03-05 01:43:46,975 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2026-03-05 01:43:46,978 - DEBUG - Ingress ctest-nginx-ingress-70249405 not present
2026-03-05 01:43:46,979 - INFO - Creating Ingress ctest-nginx-ingress-70249405
2026-03-05 01:43:46,990 - INFO - Created Ingress ctest-nginx-ingress-70249405
2026-03-05 01:43:46,993 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-70249405
2026-03-05 01:43:49,997 - DEBUG - For Ingress ctest-nginx-ingress-70249405, Cluster IP: 10.47.255.249, External IPs ['4.195.172.60']
2026-03-05 01:43:49,997 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-70249405
2026-03-05 01:43:49,997 - DEBUG - Requesting: http://10.0.0.27:8108/Snh_LoadbalancerDatabaseList?x=af41d021-288a-4dc2-956c-af293092ad5f
2026-03-05 01:43:50,003 - INFO - Ingress ctest-nginx-ingress-70249405 with uuid af41d021-288a-4dc2-956c-af293092ad5f found in kube manager
2026-03-05 01:43:50,004 - INFO - Ingress ctest-nginx-ingress-70249405 verification passed
2026-03-05 01:43:50,006 - DEBUG - Pod ctest-busybox-pod-07476265 not present
2026-03-05 01:43:50,007 - INFO - Creating Pod ctest-busybox-pod-07476265
2026-03-05 01:43:50,044 - DEBUG - Pod : ctest-busybox-pod-07476265 UUID is 1f67ed29-5e61-46f2-a242-1036db472b19
2026-03-05 01:43:50,052 - DEBUG - Pod ctest-nginx-pod-90335849 not in running state.Currently in Pending
2026-03-05 01:43:55,058 - DEBUG - Pod ctest-nginx-pod-90335849 not in running state.Currently in Pending
2026-03-05 01:44:00,065 - INFO - Pod ctest-nginx-pod-90335849 is in running state.Got IP 10.47.255.251
2026-03-05 01:44:00,126 - DEBUG - Pod ctest-nginx-pod-90335849 has vmi be9d2a58-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:44:00,126 - INFO - Verified pod ctest-nginx-pod-90335849 in contrail-api
2026-03-05 01:44:00,126 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:44:00,137 - DEBUG - VMI be9d2a58-1834-11f1-b4c4-fa163e29b957 is active in agent 10.0.0.240
2026-03-05 01:44:00,137 - INFO - Verified Pod ctest-nginx-pod-90335849 in agent 10.0.0.240
2026-03-05 01:44:00,137 - DEBUG - Requesting: http://10.0.0.27:8108/Snh_VirtualMachineDatabaseList?x=e63a95b0-06b0-41bb-be5d-d8d321e9451f
2026-03-05 01:44:00,141 - INFO - Pod ctest-nginx-pod-90335849 with uuid e63a95b0-06b0-41bb-be5d-d8d321e9451f found in kube manager
2026-03-05 01:44:00,141 - INFO - Pod ctest-nginx-pod-90335849 verification passed
2026-03-05 01:44:00,218 - DEBUG - [Pod ctest-nginx-pod-90335849] Cmd: echo ctest-nginx-pod-90335849 > /usr/share/nginx/html/index.html, Output:
2026-03-05 01:44:00,224 - INFO - Pod ctest-nginx-pod-22246185 is in running state.Got IP 10.47.255.250
2026-03-05 01:44:00,236 - DEBUG - Pod ctest-nginx-pod-22246185 has vmi bfdedce0-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:44:00,236 - INFO - Verified pod ctest-nginx-pod-22246185 in contrail-api
2026-03-05 01:44:00,236 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:44:00,245 - DEBUG - VMI bfdedce0-1834-11f1-b4c4-fa163e29b957 is active in agent 10.0.0.240
2026-03-05 01:44:00,246 - INFO - Verified Pod ctest-nginx-pod-22246185 in agent 10.0.0.240
2026-03-05 01:44:00,246 - DEBUG - Requesting: http://10.0.0.27:8108/Snh_VirtualMachineDatabaseList?x=4d676906-879d-4feb-89d4-b1b4f792c22c
2026-03-05 01:44:00,249 - INFO - Pod ctest-nginx-pod-22246185 with uuid 4d676906-879d-4feb-89d4-b1b4f792c22c found in kube manager
2026-03-05 01:44:00,249 - INFO - Pod ctest-nginx-pod-22246185 verification passed
2026-03-05 01:44:00,302 - DEBUG - [Pod ctest-nginx-pod-22246185] Cmd: echo ctest-nginx-pod-22246185 > /usr/share/nginx/html/index.html, Output:
2026-03-05 01:44:00,308 - DEBUG - Pod ctest-busybox-pod-07476265 not in running state.Currently in Pending
2026-03-05 01:44:05,321 - INFO - Pod ctest-busybox-pod-07476265 is in running state.Got IP 10.47.255.248
2026-03-05 01:44:05,332 - DEBUG - Pod ctest-busybox-pod-07476265 has vmi c243105a-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:44:05,332 - INFO - Verified pod ctest-busybox-pod-07476265 in contrail-api
2026-03-05 01:44:05,333 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:44:05,342 - DEBUG - VMI c243105a-1834-11f1-b4c4-fa163e29b957 is active in agent 10.0.0.240
2026-03-05 01:44:05,342 - INFO - Verified Pod ctest-busybox-pod-07476265 in agent 10.0.0.240
2026-03-05 01:44:05,342 - DEBUG - Requesting: http://10.0.0.27:8108/Snh_VirtualMachineDatabaseList?x=1f67ed29-5e61-46f2-a242-1036db472b19
2026-03-05 01:44:05,346 - INFO - Pod ctest-busybox-pod-07476265 with uuid 1f67ed29-5e61-46f2-a242-1036db472b19 found in kube manager
2026-03-05 01:44:05,346 - INFO - Pod ctest-busybox-pod-07476265 verification passed
2026-03-05 01:44:05,426 - DEBUG - [Pod ctest-busybox-pod-07476265] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-90335849
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2026-03-05 01:44:05,427 - DEBUG - [Pod Pod ctest-busybox-pod-07476265] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2026-03-05 01:44:05,427 - DEBUG - [Pod Pod ctest-busybox-pod-07476265] Cmd output: ctest-nginx-pod-90335849
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2026-03-05 01:44:05,489 - DEBUG - [Pod ctest-busybox-pod-07476265] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-22246185
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2026-03-05 01:44:05,489 - DEBUG - [Pod Pod ctest-busybox-pod-07476265] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2026-03-05 01:44:05,489 - DEBUG - [Pod Pod ctest-busybox-pod-07476265] Cmd output: ctest-nginx-pod-22246185
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2026-03-05 01:44:05,489 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-90335849': 1, 'ctest-nginx-pod-22246185': 1}
2026-03-05 01:44:05,489 - WARNING - Nothing to delete parallely
2026-03-05 01:44:05,490 - INFO - Deleting pod default:ctest-busybox-pod-07476265
2026-03-05 01:44:05,503 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=1f67ed29-5e61-46f2-a242-1036db472b19
2026-03-05 01:44:05,515 - WARNING - Pod uuid 1f67ed29-5e61-46f2-a242-1036db472b19 is still seen in agent 10.0.0.240 VM list
2026-03-05 01:44:10,515 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=1f67ed29-5e61-46f2-a242-1036db472b19
2026-03-05 01:44:10,523 - WARNING - Pod uuid 1f67ed29-5e61-46f2-a242-1036db472b19 is still seen in agent 10.0.0.240 VM list
2026-03-05 01:44:15,524 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=1f67ed29-5e61-46f2-a242-1036db472b19
2026-03-05 01:44:15,529 - WARNING - Pod uuid 1f67ed29-5e61-46f2-a242-1036db472b19 is still seen in agent 10.0.0.240 VM list
2026-03-05 01:44:20,530 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=1f67ed29-5e61-46f2-a242-1036db472b19
2026-03-05 01:44:20,536 - WARNING - Pod uuid 1f67ed29-5e61-46f2-a242-1036db472b19 is still seen in agent 10.0.0.240 VM list
2026-03-05 01:44:25,538 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=1f67ed29-5e61-46f2-a242-1036db472b19
2026-03-05 01:44:25,544 - WARNING - Pod uuid 1f67ed29-5e61-46f2-a242-1036db472b19 is still seen in agent 10.0.0.240 VM list
2026-03-05 01:44:30,544 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=1f67ed29-5e61-46f2-a242-1036db472b19
2026-03-05 01:44:30,550 - WARNING - Pod uuid 1f67ed29-5e61-46f2-a242-1036db472b19 is still seen in agent 10.0.0.240 VM list
2026-03-05 01:44:35,551 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=1f67ed29-5e61-46f2-a242-1036db472b19
2026-03-05 01:44:35,557 - WARNING - Pod uuid 1f67ed29-5e61-46f2-a242-1036db472b19 is still seen in agent 10.0.0.240 VM list
2026-03-05 01:44:40,557 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=1f67ed29-5e61-46f2-a242-1036db472b19
2026-03-05 01:44:40,563 - DEBUG - Pod 1f67ed29-5e61-46f2-a242-1036db472b19 is not in agent 10.0.0.240 VM list
2026-03-05 01:44:40,564 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c243105a-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:44:40,569 - DEBUG - VMI c243105a-1834-11f1-b4c4-fa163e29b957 is removed from agent 10.0.0.240
2026-03-05 01:44:40,569 - INFO - Verified that pod ctest-busybox-pod-07476265 is removed in agent
2026-03-05 01:44:40,569 - INFO - Deleting Ingress : ctest-nginx-ingress-70249405
2026-03-05 01:44:40,575 - DEBUG - Requesting: http://10.0.0.27:8108/Snh_LoadbalancerDatabaseList?x=af41d021-288a-4dc2-956c-af293092ad5f
2026-03-05 01:44:40,580 - ERROR - Ingress ctest-nginx-ingress-70249405 with uuid af41d021-288a-4dc2-956c-af293092ad5f still found in kube manager
2026-03-05 01:44:41,583 - DEBUG - Requesting: http://10.0.0.27:8108/Snh_LoadbalancerDatabaseList?x=af41d021-288a-4dc2-956c-af293092ad5f
2026-03-05 01:44:41,589 - INFO - Ingress ctest-nginx-ingress-70249405 with uuid af41d021-288a-4dc2-956c-af293092ad5f deleted successfully from kube manager
2026-03-05 01:44:41,589 - INFO - Verifications on Ingress %s cleanup passed
2026-03-05 01:44:41,589 - INFO - Deleting the FIP pool __fip_pool_public__
2026-03-05 01:44:41,705 - DEBUG - Requesting: http://10.0.0.27:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2026-03-05 01:44:41,713 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2026-03-05 01:44:41,714 - INFO - Deleting VN __public__
2026-03-05 01:44:41,808 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/1489ef8d-ec60-4c0a-a8f8-f5aebd9e008d
2026-03-05 01:44:41,814 - DEBUG - Response Code: 404
2026-03-05 01:44:41,814 - DEBUG - Requesting: http://10.0.0.27:8082/domains
2026-03-05 01:44:41,818 - DEBUG - Requesting: http://10.0.0.27:8082/domain/5725e86a-1270-4266-9169-c72cce701556
2026-03-05 01:44:41,826 - DEBUG - Requesting: http://10.0.0.27:8082/project/1034e79d-df62-4019-9087-126393eb9448
2026-03-05 01:44:41,878 - INFO - Validated that VN __public__ is not found in API Server
2026-03-05 01:44:41,878 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2026-03-05 01:44:41,885 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-03-05 01:44:41,901 - DEBUG - VN __public__ is not present in Agent 10.0.0.240
2026-03-05 01:44:41,901 - INFO - Validated that VN __public__ is not in any agent
2026-03-05 01:44:41,901 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-03-05 01:44:41,909 - DEBUG - Requesting: http://10.0.0.27:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2026-03-05 01:44:41,923 - DEBUG - Requesting: http://10.0.0.27:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2026-03-05 01:44:41,927 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2026-03-05 01:44:41,928 - INFO - Deleting pod default:ctest-nginx-pod-22246185
2026-03-05 01:44:41,937 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=4d676906-879d-4feb-89d4-b1b4f792c22c
2026-03-05 01:44:41,962 - WARNING - Pod uuid 4d676906-879d-4feb-89d4-b1b4f792c22c is still seen in agent 10.0.0.240 VM list
2026-03-05 01:44:46,963 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=4d676906-879d-4feb-89d4-b1b4f792c22c
2026-03-05 01:44:46,971 - DEBUG - Pod 4d676906-879d-4feb-89d4-b1b4f792c22c is not in agent 10.0.0.240 VM list
2026-03-05 01:44:46,971 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:bfdedce0-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:44:46,979 - DEBUG - VMI bfdedce0-1834-11f1-b4c4-fa163e29b957 is removed from agent 10.0.0.240
2026-03-05 01:44:46,979 - INFO - Verified that pod ctest-nginx-pod-22246185 is removed in agent
2026-03-05 01:44:46,979 - INFO - Deleting pod default:ctest-nginx-pod-90335849
2026-03-05 01:44:46,988 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=e63a95b0-06b0-41bb-be5d-d8d321e9451f
2026-03-05 01:44:47,008 - WARNING - Pod uuid e63a95b0-06b0-41bb-be5d-d8d321e9451f is still seen in agent 10.0.0.240 VM list
2026-03-05 01:44:52,010 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=e63a95b0-06b0-41bb-be5d-d8d321e9451f
2026-03-05 01:44:52,017 - DEBUG - Pod e63a95b0-06b0-41bb-be5d-d8d321e9451f is not in agent 10.0.0.240 VM list
2026-03-05 01:44:52,017 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:be9d2a58-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:44:52,022 - DEBUG - VMI be9d2a58-1834-11f1-b4c4-fa163e29b957 is removed from agent 10.0.0.240
2026-03-05 01:44:52,022 - INFO - Verified that pod ctest-nginx-pod-90335849 is removed in agent
2026-03-05 01:44:52,022 - INFO - Deleting service : ctest-nginx-svc-58391939
2026-03-05 01:44:52,292 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_AgentXmppConnectionStatusReq?
2026-03-05 01:44:52,297 - DEBUG - No XMPP flaps were noticed during the test
2026-03-05 01:44:52,297 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:10]
2026-03-05 01:44:52,298 - INFO - --------------------------------------------------------------------------------
2026-03-05 01:44:58,754 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2026-03-05 01:44:58,870 - DEBUG - Output : cluster_project={}
2026-03-05 01:44:58,928 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2026-03-05 01:44:59,109 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2026-03-05 01:44:59,116 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2026-03-05 01:44:59,268 - DEBUG - Output : cluster_project={}
2026-03-05 01:45:09,411 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:45:09,498 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:45:14,545 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:45:14,618 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:45:14,650 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:45:19,694 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:45:19,768 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:45:19,844 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-03-05 01:45:20,086 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_AgentXmppConnectionStatusReq?
2026-03-05 01:45:21,782 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_ApplicationPolicySetReq?
2026-03-05 01:45:21,789 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_AclReq?
2026-03-05 01:45:21,810 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_AclReq?
2026-03-05 01:45:23,832 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_AclReq?
2026-03-05 01:48:23,830 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_AgentXmppConnectionStatusReq?
2026-03-05 01:48:23,876 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=800da67e-862c-4274-aba3-bdf6ccd970a8
2026-03-05 01:48:23,895 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=aee9cb5e-835f-4742-a042-952f699dc883
2026-03-05 01:48:23,900 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=e29aabe1-6319-4139-b8ce-b2050209ce90
2026-03-05 01:48:23,904 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=7cb9911f-5e10-492a-ac5e-369090e0f645
2026-03-05 01:48:23,908 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=fd24100e-fa74-4dea-a7fa-82c0747339e3
2026-03-05 01:48:23,914 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=8991df8b-2668-486d-8fb7-10a0d6b75942
2026-03-05 01:48:23,919 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=c96dc270-95c0-44c9-834e-372657665bbe
2026-03-05 01:48:23,924 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=c3a9c236-8215-4bfb-83b6-ad5112a400fb
2026-03-05 01:48:28,957 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=c3a9c236-8215-4bfb-83b6-ad5112a400fb
2026-03-05 01:48:28,958 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=7cb9911f-5e10-492a-ac5e-369090e0f645
2026-03-05 01:48:28,959 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=800da67e-862c-4274-aba3-bdf6ccd970a8
2026-03-05 01:48:28,960 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=aee9cb5e-835f-4742-a042-952f699dc883
2026-03-05 01:48:28,961 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=e29aabe1-6319-4139-b8ce-b2050209ce90
2026-03-05 01:48:28,961 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=c96dc270-95c0-44c9-834e-372657665bbe
2026-03-05 01:48:28,962 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=fd24100e-fa74-4dea-a7fa-82c0747339e3
2026-03-05 01:48:28,963 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=8991df8b-2668-486d-8fb7-10a0d6b75942
2026-03-05 01:48:28,985 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ebe304f6-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:48:28,987 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ecdd4592-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:48:28,988 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ed5962d0-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:48:28,990 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ec53d50a-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:48:33,987 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=7cb9911f-5e10-492a-ac5e-369090e0f645
2026-03-05 01:48:33,988 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=800da67e-862c-4274-aba3-bdf6ccd970a8
2026-03-05 01:48:33,989 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=c3a9c236-8215-4bfb-83b6-ad5112a400fb
2026-03-05 01:48:33,990 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=8991df8b-2668-486d-8fb7-10a0d6b75942
2026-03-05 01:48:39,007 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=7cb9911f-5e10-492a-ac5e-369090e0f645
2026-03-05 01:48:39,008 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=c3a9c236-8215-4bfb-83b6-ad5112a400fb
2026-03-05 01:48:39,009 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=8991df8b-2668-486d-8fb7-10a0d6b75942
2026-03-05 01:48:39,011 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=800da67e-862c-4274-aba3-bdf6ccd970a8
2026-03-05 01:48:44,019 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=7cb9911f-5e10-492a-ac5e-369090e0f645
2026-03-05 01:48:44,021 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=800da67e-862c-4274-aba3-bdf6ccd970a8
2026-03-05 01:48:44,022 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=8991df8b-2668-486d-8fb7-10a0d6b75942
2026-03-05 01:48:44,023 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=c3a9c236-8215-4bfb-83b6-ad5112a400fb
2026-03-05 01:48:49,033 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=800da67e-862c-4274-aba3-bdf6ccd970a8
2026-03-05 01:48:49,034 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=c3a9c236-8215-4bfb-83b6-ad5112a400fb
2026-03-05 01:48:49,035 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=8991df8b-2668-486d-8fb7-10a0d6b75942
2026-03-05 01:48:49,036 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=7cb9911f-5e10-492a-ac5e-369090e0f645
2026-03-05 01:48:54,050 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=7cb9911f-5e10-492a-ac5e-369090e0f645
2026-03-05 01:48:54,052 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=c3a9c236-8215-4bfb-83b6-ad5112a400fb
2026-03-05 01:48:54,053 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=8991df8b-2668-486d-8fb7-10a0d6b75942
2026-03-05 01:48:54,053 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=800da67e-862c-4274-aba3-bdf6ccd970a8
2026-03-05 01:48:59,095 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=7cb9911f-5e10-492a-ac5e-369090e0f645
2026-03-05 01:48:59,098 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=c3a9c236-8215-4bfb-83b6-ad5112a400fb
2026-03-05 01:48:59,099 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=8991df8b-2668-486d-8fb7-10a0d6b75942
2026-03-05 01:48:59,100 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=800da67e-862c-4274-aba3-bdf6ccd970a8
2026-03-05 01:48:59,102 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ede7e78a-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:48:59,112 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ee59cfbc-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:48:59,113 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ef4bdfe6-1834-11f1-b4c4-fa163e29b957
2026-03-05 01:48:59,115 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:eecd2778-1834-11f1-b4c4-fa163e29b957