2025-10-21 21:15:50,601 - DEBUG - [10.0.0.72]: Running cmd : hostname
2025-10-21 21:15:51,081 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1586-1
2025-10-21 21:15:51,081 - DEBUG - [10.0.0.72]: Running cmd : hostname -f
2025-10-21 21:15:51,185 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1586-1
2025-10-21 21:15:51,185 - DEBUG - [10.0.0.72]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-10-21 21:15:51,384 - DEBUG - Output : NAMES
contrail_test_qNSBkraqI
contrail-kubernetes-master-kubemanager-1
analytics_database-query-engine-1
analytics_database-nodemgr-1
analytics_database-cassandra-1
analytics_snmp-snmp-collector-1
analytics_snmp-topology-1
analytics_snmp-nodemgr-1
analytics_alarm-alarm-gen-1
analytics_alarm-kafka-1
analytics_alarm-nodemgr-1
analytics-nodemgr-1
analytics-api-1
analytics-collector-1
redis-redis-1
webui-job-1
webui-web-1
control-named-1
control-control-1
control-nodemgr-1
control-dns-1
config_database-rabbitmq-1
config_database-nodemgr-1
config_database-zookeeper-1
config_database-cassandra-1
config_api-devicemgr-1
config_api-svcmonitor-1
config_api-api-1
config_api-dnsmasq-1
config_api-nodemgr-1
config_api-schema-1
2025-10-21 21:15:51,386 - DEBUG - [10.0.0.72]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-10-21 21:15:51,477 - DEBUG - Output : 127.0.0.1/8
10.0.0.72/24
252.72.0.1/8
10.135.235.1/24
172.17.0.1/16
2025-10-21 21:15:51,478 - DEBUG - [10.0.0.72]: Running cmd : getent hosts 10.0.0.72 | head -n 1 | awk '{print $2}'
2025-10-21 21:15:51,603 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1586-1
2025-10-21 21:15:51,788 - DEBUG - [10.0.0.136]: Running cmd : hostname
2025-10-21 21:15:52,041 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1586-1
2025-10-21 21:15:52,041 - DEBUG - [10.0.0.136]: Running cmd : hostname -f
2025-10-21 21:15:52,079 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1586-1
2025-10-21 21:15:52,080 - DEBUG - [10.0.0.136]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-10-21 21:15:52,161 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2025-10-21 21:15:52,162 - DEBUG - [10.0.0.136]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-10-21 21:15:52,200 - DEBUG - Output : 10.0.0.136/24
2025-10-21 21:15:52,201 - DEBUG - [10.0.0.136]: Running cmd : getent hosts 10.0.0.136 | head -n 1 | awk '{print $2}'
2025-10-21 21:15:52,273 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1586-1
2025-10-21 21:15:52,387 - INFO - Using existing project ['default-domain', 'k8s-default'](93875a54-76cd-4b69-a1be-fd66a3f0eda3)
2025-10-21 21:15:52,462 - INFO - Using existing project ['default-domain', 'k8s-default'](93875a54-76cd-4b69-a1be-fd66a3f0eda3)
2025-10-21 21:15:52,565 - INFO - ================================================================================
2025-10-21 21:15:52,565 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-10-21 21:15:52,565 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-10-21 21:15:52,848 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_AgentXmppConnectionStatusReq?
2025-10-21 21:15:52,862 - DEBUG - Nothing to compare xmpp stats {'10.0.0.136': {'10.0.0.72': '0'}} with
2025-10-21 21:15:52,862 - INFO - Initial checks done. Running the testcase now
2025-10-21 21:15:52,862 - INFO -
2025-10-21 21:15:52,873 - INFO - Namespace default already exists
2025-10-21 21:15:52,873 - DEBUG - [10.0.0.72]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-10-21 21:15:53,143 - DEBUG - Output : cluster_project={}
2025-10-21 21:15:53,269 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-10-21 21:15:53,269 - DEBUG - Project uuid in Contrail is 93875a54-76cd-4b69-a1be-fd66a3f0eda3
2025-10-21 21:15:53,270 - DEBUG - [10.0.0.72]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-10-21 21:15:53,481 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-10-21 21:15:53,481 - DEBUG - Requesting: http://10.0.0.72:8108/Snh_NamespaceDatabaseList?namespace_uuid=9348bde4-7e96-4a40-b578-4f3662727cf2
2025-10-21 21:15:53,488 - INFO - Namespace default with uuid 9348bde4-7e96-4a40-b578-4f3662727cf2 found in kube manager
2025-10-21 21:15:53,488 - INFO - Namespace default verification passed
2025-10-21 21:15:53,493 - DEBUG - Service ctest-nginx-svc-01121754 not present
2025-10-21 21:15:53,494 - INFO - Creating service ctest-nginx-svc-01121754
2025-10-21 21:15:53,518 - INFO - Created Service ctest-nginx-svc-01121754
2025-10-21 21:15:53,535 - DEBUG - Pod ctest-nginx-pod-64481119 not present
2025-10-21 21:15:53,536 - INFO - Creating Pod ctest-nginx-pod-64481119
2025-10-21 21:15:53,615 - DEBUG - Node for Pod ctest-nginx-pod-64481119 not yet populated
2025-10-21 21:15:56,635 - DEBUG - Pod : ctest-nginx-pod-64481119 UUID is 560c150f-0c7f-43ae-b07b-f0287a914f21
2025-10-21 21:15:56,640 - DEBUG - Pod ctest-nginx-pod-34524892 not present
2025-10-21 21:15:56,641 - INFO - Creating Pod ctest-nginx-pod-34524892
2025-10-21 21:15:56,707 - DEBUG - Pod : ctest-nginx-pod-34524892 UUID is 6a1359f5-ae55-48bd-856e-4b643065921a
2025-10-21 21:15:56,937 - INFO - Created VN __public__, UUID :d6316d12-cece-4922-a470-03c37a1e1217
2025-10-21 21:15:57,151 - DEBUG - Requesting: http://10.0.0.72:8082/domains
2025-10-21 21:15:57,156 - DEBUG - Requesting: http://10.0.0.72:8082/domain/b072918b-e702-4761-a8f6-d1c129fa9bfc
2025-10-21 21:15:57,169 - DEBUG - Requesting: http://10.0.0.72:8082/project/93875a54-76cd-4b69-a1be-fd66a3f0eda3
2025-10-21 21:15:57,216 - DEBUG - Requesting: http://10.0.0.72:8082/virtual-network/d6316d12-cece-4922-a470-03c37a1e1217
2025-10-21 21:15:57,239 - DEBUG - Requesting: http://10.0.0.72:8082/virtual-network/d6316d12-cece-4922-a470-03c37a1e1217
2025-10-21 21:15:57,247 - DEBUG - Requesting: http://10.0.0.72:8082/routing-instance/be4c8912-28d8-465a-808d-d096606ede4e
2025-10-21 21:15:57,260 - DEBUG - Requesting: http://10.0.0.72:8082/routing-instance/be4c8912-28d8-465a-808d-d096606ede4e
2025-10-21 21:15:57,268 - DEBUG - Requesting: http://10.0.0.72:8082/route-target/df0abd58-4654-465f-b600-9cfceb44281c
2025-10-21 21:15:57,277 - DEBUG - Route Targets: ['target:64512:8000006']
2025-10-21 21:15:57,277 - DEBUG - Requesting: http://10.0.0.72:8082/virtual-network/d6316d12-cece-4922-a470-03c37a1e1217
2025-10-21 21:15:57,284 - DEBUG - Requesting: http://10.0.0.72:8082/routing-instance/be4c8912-28d8-465a-808d-d096606ede4e
2025-10-21 21:15:57,298 - INFO - Verified VN network id 13 for VN d6316d12-cece-4922-a470-03c37a1e1217
2025-10-21 21:15:57,298 - INFO - Verifications in API Server for VN __public__ passed
2025-10-21 21:15:57,298 - DEBUG - Requesting: http://10.0.0.72:8082/virtual-network/d6316d12-cece-4922-a470-03c37a1e1217
2025-10-21 21:15:57,305 - DEBUG - Requesting: http://10.0.0.72:8082/routing-instance/be4c8912-28d8-465a-808d-d096606ede4e
2025-10-21 21:15:57,309 - DEBUG - Requesting: http://10.0.0.72:8082/routing-instance/be4c8912-28d8-465a-808d-d096606ede4e
2025-10-21 21:15:57,315 - DEBUG - Requesting: http://10.0.0.72:8082/route-target/df0abd58-4654-465f-b600-9cfceb44281c
2025-10-21 21:15:57,319 - DEBUG - Requesting: http://10.0.0.72:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-10-21 21:15:57,324 - DEBUG - Control-node 10.0.0.72 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15434237325521864994', 'uuid-lslong': '11848974757714072087'}, 'enable': 'true', 'created': '2025-10-21T21:15:56', 'last-modified': '2025-10-21T21:15:57', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.306128'}
2025-10-21 21:15:57,324 - DEBUG - Requesting: http://10.0.0.72:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-10-21 21:15:57,349 - DEBUG - Route Targets: ['target:64512:8000006']
2025-10-21 21:15:57,349 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-10-21 21:15:57,350 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-10-21 21:15:57,350 - DEBUG - Requesting: http://10.0.0.72:8082/domains
2025-10-21 21:15:57,363 - DEBUG - Requesting: http://10.0.0.72:8082/domain/b072918b-e702-4761-a8f6-d1c129fa9bfc
2025-10-21 21:15:57,369 - DEBUG - Requesting: http://10.0.0.72:8082/project/93875a54-76cd-4b69-a1be-fd66a3f0eda3
2025-10-21 21:15:57,409 - DEBUG - Requesting: http://10.0.0.72:8082/virtual-network/d6316d12-cece-4922-a470-03c37a1e1217
2025-10-21 21:15:57,427 - DEBUG - =>VN __public__ has no policy to be verified
2025-10-21 21:15:57,427 - DEBUG - Verifying the vn in opserver
2025-10-21 21:15:57,427 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.72
2025-10-21 21:15:57,428 - DEBUG - Requesting: http://10.0.0.72:8081/analytics/uves/virtual-networks
2025-10-21 21:15:57,432 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.72:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-10-21 21:15:57,432 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-10-21 21:15:57,433 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-10-21 21:15:57,440 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-10-21 21:15:57,448 - DEBUG - Do not have enough data to verify VN in agent
2025-10-21 21:15:57,448 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-10-21 21:15:57,455 - DEBUG - VRF ids for VN __public__: {}
2025-10-21 21:15:57,553 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-10-21 21:15:57,647 - DEBUG - Requesting: http://10.0.0.72:8082/domains
2025-10-21 21:15:57,651 - DEBUG - Requesting: http://10.0.0.72:8082/domain/b072918b-e702-4761-a8f6-d1c129fa9bfc
2025-10-21 21:15:57,657 - DEBUG - Requesting: http://10.0.0.72:8082/project/93875a54-76cd-4b69-a1be-fd66a3f0eda3
2025-10-21 21:15:57,696 - DEBUG - Requesting: http://10.0.0.72:8082/virtual-network/d6316d12-cece-4922-a470-03c37a1e1217
2025-10-21 21:15:57,710 - DEBUG - Requesting: http://10.0.0.72:8082/floating-ip-pool/e79fe1da-fdcb-4717-889f-a467980cb032
2025-10-21 21:15:57,715 - DEBUG - Requesting: http://10.0.0.72:8082/domains
2025-10-21 21:15:57,720 - DEBUG - Requesting: http://10.0.0.72:8082/domain/b072918b-e702-4761-a8f6-d1c129fa9bfc
2025-10-21 21:15:57,725 - DEBUG - Requesting: http://10.0.0.72:8082/project/93875a54-76cd-4b69-a1be-fd66a3f0eda3
2025-10-21 21:15:57,762 - DEBUG - Requesting: http://10.0.0.72:8082/virtual-network/d6316d12-cece-4922-a470-03c37a1e1217
2025-10-21 21:15:57,770 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-10-21 21:15:57,770 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-10-21 21:15:57,777 - DEBUG - Requesting: http://10.0.0.72:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-10-21 21:15:57,783 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-10-21 21:15:57,783 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-10-21 21:15:57,786 - DEBUG - Ingress ctest-nginx-ingress-98957996 not present
2025-10-21 21:15:57,787 - INFO - Creating Ingress ctest-nginx-ingress-98957996
2025-10-21 21:15:57,798 - INFO - Created Ingress ctest-nginx-ingress-98957996
2025-10-21 21:15:57,802 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-98957996
2025-10-21 21:16:00,807 - DEBUG - For Ingress ctest-nginx-ingress-98957996, Cluster IP: 10.47.255.249, External IPs ['81.213.184.252']
2025-10-21 21:16:00,807 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-98957996
2025-10-21 21:16:00,807 - DEBUG - Requesting: http://10.0.0.72:8108/Snh_LoadbalancerDatabaseList?x=b55ea418-b74d-444b-9527-3dc057f8c422
2025-10-21 21:16:00,813 - INFO - Ingress ctest-nginx-ingress-98957996 with uuid b55ea418-b74d-444b-9527-3dc057f8c422 found in kube manager
2025-10-21 21:16:00,813 - INFO - Ingress ctest-nginx-ingress-98957996 verification passed
2025-10-21 21:16:00,816 - DEBUG - Pod ctest-busybox-pod-83093166 not present
2025-10-21 21:16:00,816 - INFO - Creating Pod ctest-busybox-pod-83093166
2025-10-21 21:16:00,835 - DEBUG - Pod : ctest-busybox-pod-83093166 UUID is 5212c682-85ce-4191-b85b-836bcbccc4a5
2025-10-21 21:16:00,852 - DEBUG - Pod ctest-nginx-pod-64481119 not in running state.Currently in Pending
2025-10-21 21:16:05,866 - INFO - Pod ctest-nginx-pod-64481119 is in running state.Got IP 10.47.255.251
2025-10-21 21:16:05,905 - DEBUG - Pod ctest-nginx-pod-64481119 has vmi 215ed96a-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:16:05,905 - INFO - Verified pod ctest-nginx-pod-64481119 in contrail-api
2025-10-21 21:16:05,905 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:16:05,937 - DEBUG - VMI 215ed96a-aec3-11f0-ac49-fa163e205ff9 is active in agent 10.0.0.136
2025-10-21 21:16:05,937 - INFO - Verified Pod ctest-nginx-pod-64481119 in agent 10.0.0.136
2025-10-21 21:16:05,937 - DEBUG - Requesting: http://10.0.0.72:8108/Snh_VirtualMachineDatabaseList?x=560c150f-0c7f-43ae-b07b-f0287a914f21
2025-10-21 21:16:05,943 - INFO - Pod ctest-nginx-pod-64481119 with uuid 560c150f-0c7f-43ae-b07b-f0287a914f21 found in kube manager
2025-10-21 21:16:05,944 - INFO - Pod ctest-nginx-pod-64481119 verification passed
2025-10-21 21:16:06,029 - DEBUG - [Pod ctest-nginx-pod-64481119] Cmd: echo ctest-nginx-pod-64481119 > /usr/share/nginx/html/index.html, Output:
2025-10-21 21:16:06,034 - DEBUG - Pod ctest-nginx-pod-34524892 not in running state.Currently in Pending
2025-10-21 21:16:11,052 - INFO - Pod ctest-nginx-pod-34524892 is in running state.Got IP 10.47.255.250
2025-10-21 21:16:11,125 - DEBUG - Pod ctest-nginx-pod-34524892 has vmi 22802042-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:16:11,125 - INFO - Verified pod ctest-nginx-pod-34524892 in contrail-api
2025-10-21 21:16:11,125 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:16:11,141 - DEBUG - VMI 22802042-aec3-11f0-ac49-fa163e205ff9 is active in agent 10.0.0.136
2025-10-21 21:16:11,141 - INFO - Verified Pod ctest-nginx-pod-34524892 in agent 10.0.0.136
2025-10-21 21:16:11,141 - DEBUG - Requesting: http://10.0.0.72:8108/Snh_VirtualMachineDatabaseList?x=6a1359f5-ae55-48bd-856e-4b643065921a
2025-10-21 21:16:11,147 - INFO - Pod ctest-nginx-pod-34524892 with uuid 6a1359f5-ae55-48bd-856e-4b643065921a found in kube manager
2025-10-21 21:16:11,148 - INFO - Pod ctest-nginx-pod-34524892 verification passed
2025-10-21 21:16:11,226 - DEBUG - [Pod ctest-nginx-pod-34524892] Cmd: echo ctest-nginx-pod-34524892 > /usr/share/nginx/html/index.html, Output:
2025-10-21 21:16:11,241 - INFO - Pod ctest-busybox-pod-83093166 is in running state.Got IP 10.47.255.248
2025-10-21 21:16:11,261 - DEBUG - Pod ctest-busybox-pod-83093166 has vmi 24f43322-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:16:11,261 - INFO - Verified pod ctest-busybox-pod-83093166 in contrail-api
2025-10-21 21:16:11,261 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:16:11,272 - DEBUG - VMI 24f43322-aec3-11f0-ac49-fa163e205ff9 is active in agent 10.0.0.136
2025-10-21 21:16:11,272 - INFO - Verified Pod ctest-busybox-pod-83093166 in agent 10.0.0.136
2025-10-21 21:16:11,272 - DEBUG - Requesting: http://10.0.0.72:8108/Snh_VirtualMachineDatabaseList?x=5212c682-85ce-4191-b85b-836bcbccc4a5
2025-10-21 21:16:11,277 - INFO - Pod ctest-busybox-pod-83093166 with uuid 5212c682-85ce-4191-b85b-836bcbccc4a5 found in kube manager
2025-10-21 21:16:11,277 - INFO - Pod ctest-busybox-pod-83093166 verification passed
2025-10-21 21:16:11,348 - DEBUG - [Pod ctest-busybox-pod-83093166] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-34524892
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-10-21 21:16:11,348 - DEBUG - [Pod Pod ctest-busybox-pod-83093166] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-10-21 21:16:11,348 - DEBUG - [Pod Pod ctest-busybox-pod-83093166] Cmd output: ctest-nginx-pod-34524892
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-10-21 21:16:11,410 - DEBUG - [Pod ctest-busybox-pod-83093166] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-64481119
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-10-21 21:16:11,410 - DEBUG - [Pod Pod ctest-busybox-pod-83093166] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-10-21 21:16:11,410 - DEBUG - [Pod Pod ctest-busybox-pod-83093166] Cmd output: ctest-nginx-pod-64481119
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-10-21 21:16:11,410 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-64481119': 1, 'ctest-nginx-pod-34524892': 1}
2025-10-21 21:16:11,411 - WARNING - Nothing to delete parallely
2025-10-21 21:16:11,411 - INFO - Deleting pod default:ctest-busybox-pod-83093166
2025-10-21 21:16:11,435 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=5212c682-85ce-4191-b85b-836bcbccc4a5
2025-10-21 21:16:11,444 - WARNING - Pod uuid 5212c682-85ce-4191-b85b-836bcbccc4a5 is still seen in agent 10.0.0.136 VM list
2025-10-21 21:16:16,445 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=5212c682-85ce-4191-b85b-836bcbccc4a5
2025-10-21 21:16:16,457 - WARNING - Pod uuid 5212c682-85ce-4191-b85b-836bcbccc4a5 is still seen in agent 10.0.0.136 VM list
2025-10-21 21:16:21,458 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=5212c682-85ce-4191-b85b-836bcbccc4a5
2025-10-21 21:16:21,470 - WARNING - Pod uuid 5212c682-85ce-4191-b85b-836bcbccc4a5 is still seen in agent 10.0.0.136 VM list
2025-10-21 21:16:26,471 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=5212c682-85ce-4191-b85b-836bcbccc4a5
2025-10-21 21:16:26,485 - WARNING - Pod uuid 5212c682-85ce-4191-b85b-836bcbccc4a5 is still seen in agent 10.0.0.136 VM list
2025-10-21 21:16:31,486 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=5212c682-85ce-4191-b85b-836bcbccc4a5
2025-10-21 21:16:31,498 - WARNING - Pod uuid 5212c682-85ce-4191-b85b-836bcbccc4a5 is still seen in agent 10.0.0.136 VM list
2025-10-21 21:16:36,500 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=5212c682-85ce-4191-b85b-836bcbccc4a5
2025-10-21 21:16:36,516 - WARNING - Pod uuid 5212c682-85ce-4191-b85b-836bcbccc4a5 is still seen in agent 10.0.0.136 VM list
2025-10-21 21:16:41,519 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=5212c682-85ce-4191-b85b-836bcbccc4a5
2025-10-21 21:16:41,531 - WARNING - Pod uuid 5212c682-85ce-4191-b85b-836bcbccc4a5 is still seen in agent 10.0.0.136 VM list
2025-10-21 21:16:46,531 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=5212c682-85ce-4191-b85b-836bcbccc4a5
2025-10-21 21:16:46,538 - DEBUG - Pod 5212c682-85ce-4191-b85b-836bcbccc4a5 is not in agent 10.0.0.136 VM list
2025-10-21 21:16:46,538 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:24f43322-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:16:46,544 - DEBUG - VMI 24f43322-aec3-11f0-ac49-fa163e205ff9 is removed from agent 10.0.0.136
2025-10-21 21:16:46,544 - INFO - Verified that pod ctest-busybox-pod-83093166 is removed in agent
2025-10-21 21:16:46,544 - INFO - Deleting Ingress : ctest-nginx-ingress-98957996
2025-10-21 21:16:46,561 - DEBUG - Requesting: http://10.0.0.72:8108/Snh_LoadbalancerDatabaseList?x=b55ea418-b74d-444b-9527-3dc057f8c422
2025-10-21 21:16:46,565 - ERROR - Ingress ctest-nginx-ingress-98957996 with uuid b55ea418-b74d-444b-9527-3dc057f8c422 still found in kube manager
2025-10-21 21:16:47,566 - DEBUG - Requesting: http://10.0.0.72:8108/Snh_LoadbalancerDatabaseList?x=b55ea418-b74d-444b-9527-3dc057f8c422
2025-10-21 21:16:47,574 - ERROR - Ingress ctest-nginx-ingress-98957996 with uuid b55ea418-b74d-444b-9527-3dc057f8c422 still found in kube manager
2025-10-21 21:16:48,575 - DEBUG - Requesting: http://10.0.0.72:8108/Snh_LoadbalancerDatabaseList?x=b55ea418-b74d-444b-9527-3dc057f8c422
2025-10-21 21:16:48,582 - INFO - Ingress ctest-nginx-ingress-98957996 with uuid b55ea418-b74d-444b-9527-3dc057f8c422 deleted successfully from kube manager
2025-10-21 21:16:48,583 - INFO - Verifications on Ingress %s cleanup passed
2025-10-21 21:16:48,583 - INFO - Deleting the FIP pool __fip_pool_public__
2025-10-21 21:16:48,684 - DEBUG - Requesting: http://10.0.0.72:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-10-21 21:16:48,691 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-10-21 21:16:48,691 - INFO - Deleting VN __public__
2025-10-21 21:16:48,832 - DEBUG - Requesting: http://10.0.0.72:8082/routing-instance/be4c8912-28d8-465a-808d-d096606ede4e
2025-10-21 21:16:48,838 - DEBUG - Response Code: 404
2025-10-21 21:16:48,838 - DEBUG - Requesting: http://10.0.0.72:8082/domains
2025-10-21 21:16:48,842 - DEBUG - Requesting: http://10.0.0.72:8082/domain/b072918b-e702-4761-a8f6-d1c129fa9bfc
2025-10-21 21:16:48,847 - DEBUG - Requesting: http://10.0.0.72:8082/project/93875a54-76cd-4b69-a1be-fd66a3f0eda3
2025-10-21 21:16:48,891 - INFO - Validated that VN __public__ is not found in API Server
2025-10-21 21:16:48,891 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-10-21 21:16:48,901 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-10-21 21:16:48,914 - DEBUG - VN __public__ is not present in Agent 10.0.0.136
2025-10-21 21:16:48,914 - INFO - Validated that VN __public__ is not in any agent
2025-10-21 21:16:48,914 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-10-21 21:16:48,921 - DEBUG - Requesting: http://10.0.0.72:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-10-21 21:16:48,925 - DEBUG - Requesting: http://10.0.0.72:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-10-21 21:16:48,932 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-10-21 21:16:48,936 - INFO - Deleting pod default:ctest-nginx-pod-34524892
2025-10-21 21:16:48,953 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=6a1359f5-ae55-48bd-856e-4b643065921a
2025-10-21 21:16:48,961 - WARNING - Pod uuid 6a1359f5-ae55-48bd-856e-4b643065921a is still seen in agent 10.0.0.136 VM list
2025-10-21 21:16:53,963 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=6a1359f5-ae55-48bd-856e-4b643065921a
2025-10-21 21:16:53,974 - DEBUG - Pod 6a1359f5-ae55-48bd-856e-4b643065921a is not in agent 10.0.0.136 VM list
2025-10-21 21:16:53,974 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:22802042-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:16:53,985 - DEBUG - VMI 22802042-aec3-11f0-ac49-fa163e205ff9 is removed from agent 10.0.0.136
2025-10-21 21:16:53,986 - INFO - Verified that pod ctest-nginx-pod-34524892 is removed in agent
2025-10-21 21:16:53,987 - INFO - Deleting pod default:ctest-nginx-pod-64481119
2025-10-21 21:16:54,038 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=560c150f-0c7f-43ae-b07b-f0287a914f21
2025-10-21 21:16:54,053 - WARNING - Pod uuid 560c150f-0c7f-43ae-b07b-f0287a914f21 is still seen in agent 10.0.0.136 VM list
2025-10-21 21:16:59,054 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=560c150f-0c7f-43ae-b07b-f0287a914f21
2025-10-21 21:16:59,065 - DEBUG - Pod 560c150f-0c7f-43ae-b07b-f0287a914f21 is not in agent 10.0.0.136 VM list
2025-10-21 21:16:59,065 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:215ed96a-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:16:59,074 - DEBUG - VMI 215ed96a-aec3-11f0-ac49-fa163e205ff9 is removed from agent 10.0.0.136
2025-10-21 21:16:59,074 - INFO - Verified that pod ctest-nginx-pod-64481119 is removed in agent
2025-10-21 21:16:59,075 - INFO - Deleting service : ctest-nginx-svc-01121754
2025-10-21 21:16:59,392 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_AgentXmppConnectionStatusReq?
2025-10-21 21:16:59,403 - DEBUG - No XMPP flaps were noticed during the test
2025-10-21 21:16:59,404 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:07]
2025-10-21 21:16:59,404 - INFO - --------------------------------------------------------------------------------
2025-10-21 21:17:09,087 - DEBUG - [10.0.0.72]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-10-21 21:17:09,226 - DEBUG - Output : cluster_project={}
2025-10-21 21:17:09,282 - DEBUG - [10.0.0.72]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-10-21 21:17:09,426 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-10-21 21:17:09,430 - DEBUG - [10.0.0.72]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-10-21 21:17:09,595 - DEBUG - Output : cluster_project={}
2025-10-21 21:17:14,711 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:17:24,807 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:17:24,881 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:17:24,986 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:17:25,076 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:17:25,151 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:17:25,282 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:17:25,364 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-21 21:17:25,689 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_AgentXmppConnectionStatusReq?
2025-10-21 21:17:28,885 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_ApplicationPolicySetReq?
2025-10-21 21:17:28,897 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_AclReq?
2025-10-21 21:17:28,946 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_AclReq?
2025-10-21 21:17:31,005 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_AclReq?
2025-10-21 21:20:32,248 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_AgentXmppConnectionStatusReq?
2025-10-21 21:20:32,344 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=e4247939-5dcf-464a-8c4b-45d8accbcfc9
2025-10-21 21:20:32,352 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=0af9eeab-7f98-4e91-92a5-7dbc52971175
2025-10-21 21:20:32,378 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=817a3694-7006-481a-b07c-ae617b24a058
2025-10-21 21:20:32,382 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=2e11481e-f55b-4869-be68-eb612fdd180f
2025-10-21 21:20:32,389 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=2bbf07b6-5180-4043-b863-dc665181b061
2025-10-21 21:20:32,396 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=e7308311-aa41-4deb-9d62-286c42f6500a
2025-10-21 21:20:32,401 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=399b671d-10b1-4882-8f49-c239101da2ee
2025-10-21 21:20:32,405 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=9747602f-2670-4d3b-ad05-8d9f9c60f2a4
2025-10-21 21:20:37,409 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=0af9eeab-7f98-4e91-92a5-7dbc52971175
2025-10-21 21:20:37,411 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=e4247939-5dcf-464a-8c4b-45d8accbcfc9
2025-10-21 21:20:37,412 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=2bbf07b6-5180-4043-b863-dc665181b061
2025-10-21 21:20:37,414 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=817a3694-7006-481a-b07c-ae617b24a058
2025-10-21 21:20:37,415 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=2e11481e-f55b-4869-be68-eb612fdd180f
2025-10-21 21:20:37,434 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4eaff0e8-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:20:37,439 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=9747602f-2670-4d3b-ad05-8d9f9c60f2a4
2025-10-21 21:20:37,441 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=399b671d-10b1-4882-8f49-c239101da2ee
2025-10-21 21:20:37,444 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=e7308311-aa41-4deb-9d62-286c42f6500a
2025-10-21 21:20:37,460 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4c6a2f42-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:20:37,486 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4d7fa43e-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:20:37,490 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4e386c58-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:20:42,437 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=0af9eeab-7f98-4e91-92a5-7dbc52971175
2025-10-21 21:20:42,458 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=817a3694-7006-481a-b07c-ae617b24a058
2025-10-21 21:20:42,462 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=2e11481e-f55b-4869-be68-eb612fdd180f
2025-10-21 21:20:42,486 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=9747602f-2670-4d3b-ad05-8d9f9c60f2a4
2025-10-21 21:20:47,449 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=0af9eeab-7f98-4e91-92a5-7dbc52971175
2025-10-21 21:20:47,476 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=2e11481e-f55b-4869-be68-eb612fdd180f
2025-10-21 21:20:47,482 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=817a3694-7006-481a-b07c-ae617b24a058
2025-10-21 21:20:47,500 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=9747602f-2670-4d3b-ad05-8d9f9c60f2a4
2025-10-21 21:20:52,460 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=0af9eeab-7f98-4e91-92a5-7dbc52971175
2025-10-21 21:20:52,499 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=817a3694-7006-481a-b07c-ae617b24a058
2025-10-21 21:20:52,502 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=2e11481e-f55b-4869-be68-eb612fdd180f
2025-10-21 21:20:52,512 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=9747602f-2670-4d3b-ad05-8d9f9c60f2a4
2025-10-21 21:20:57,475 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=0af9eeab-7f98-4e91-92a5-7dbc52971175
2025-10-21 21:20:57,519 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=817a3694-7006-481a-b07c-ae617b24a058
2025-10-21 21:20:57,523 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=2e11481e-f55b-4869-be68-eb612fdd180f
2025-10-21 21:20:57,529 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=9747602f-2670-4d3b-ad05-8d9f9c60f2a4
2025-10-21 21:21:02,488 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=0af9eeab-7f98-4e91-92a5-7dbc52971175
2025-10-21 21:21:02,537 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=817a3694-7006-481a-b07c-ae617b24a058
2025-10-21 21:21:02,541 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=2e11481e-f55b-4869-be68-eb612fdd180f
2025-10-21 21:21:02,544 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=9747602f-2670-4d3b-ad05-8d9f9c60f2a4
2025-10-21 21:21:07,501 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=0af9eeab-7f98-4e91-92a5-7dbc52971175
2025-10-21 21:21:07,509 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4fa54cbe-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:21:07,547 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=2e11481e-f55b-4869-be68-eb612fdd180f
2025-10-21 21:21:07,549 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=817a3694-7006-481a-b07c-ae617b24a058
2025-10-21 21:21:07,551 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_VmListReq?uuid=9747602f-2670-4d3b-ad05-8d9f9c60f2a4
2025-10-21 21:21:07,558 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:50176ed4-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:21:07,559 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:50833f7e-aec3-11f0-ac49-fa163e205ff9
2025-10-21 21:21:07,563 - DEBUG - Requesting: http://10.0.0.136:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4f2d2356-aec3-11f0-ac49-fa163e205ff9