2026-02-09 01:50:16,058 - DEBUG - [10.0.0.129]: Running cmd : hostname
2026-02-09 01:50:16,307 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1861-1
2026-02-09 01:50:16,308 - DEBUG - [10.0.0.129]: Running cmd : hostname -f
2026-02-09 01:50:16,341 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1861-1
2026-02-09 01:50:16,341 - DEBUG - [10.0.0.129]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2026-02-09 01:50:16,453 - DEBUG - Output : NAMES
contrail_test_MptAGeqlx
analytics_database-nodemgr-1
analytics_database-cassandra-1
analytics_database-query-engine-1
analytics_snmp-snmp-collector-1
analytics_snmp-nodemgr-1
analytics_snmp-topology-1
analytics_alarm-kafka-1
analytics_alarm-nodemgr-1
analytics_alarm-alarm-gen-1
analytics-collector-1
analytics-api-1
analytics-nodemgr-1
contrail-kubernetes-master-kubemanager-1
redis-redis-1
webui-job-1
webui-web-1
control-named-1
control-dns-1
control-nodemgr-1
control-control-1
config_database-cassandra-1
config_database-zookeeper-1
config_database-rabbitmq-1
config_database-nodemgr-1
config_api-devicemgr-1
config_api-dnsmasq-1
config_api-svcmonitor-1
config_api-api-1
config_api-nodemgr-1
config_api-schema-1
2026-02-09 01:50:16,454 - DEBUG - [10.0.0.129]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2026-02-09 01:50:16,526 - DEBUG - Output : 127.0.0.1/8
10.0.0.129/24
252.129.0.1/8
10.21.133.1/24
172.17.0.1/16
2026-02-09 01:50:16,526 - DEBUG - [10.0.0.129]: Running cmd : getent hosts 10.0.0.129 | head -n 1 | awk '{print $2}'
2026-02-09 01:50:16,611 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1861-1
2026-02-09 01:50:16,716 - DEBUG - [10.0.0.242]: Running cmd : hostname
2026-02-09 01:50:16,990 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1861-1
2026-02-09 01:50:16,990 - DEBUG - [10.0.0.242]: Running cmd : hostname -f
2026-02-09 01:50:17,047 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1861-1
2026-02-09 01:50:17,047 - DEBUG - [10.0.0.242]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2026-02-09 01:50:17,093 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2026-02-09 01:50:17,093 - DEBUG - [10.0.0.242]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2026-02-09 01:50:17,128 - DEBUG - Output : 10.0.0.242/24
2026-02-09 01:50:17,128 - DEBUG - [10.0.0.242]: Running cmd : getent hosts 10.0.0.242 | head -n 1 | awk '{print $2}'
2026-02-09 01:50:17,196 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1861-1
2026-02-09 01:50:17,300 - INFO - Using existing project ['default-domain', 'k8s-default'](668100c7-bb7d-408d-bb3e-363efce8b4b2)
2026-02-09 01:50:17,406 - INFO - Using existing project ['default-domain', 'k8s-default'](668100c7-bb7d-408d-bb3e-363efce8b4b2)
2026-02-09 01:50:17,544 - INFO - ================================================================================
2026-02-09 01:50:17,544 - INFO - STARTING TEST : test_ingress_ip_assignment
2026-02-09 01:50:17,544 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2026-02-09 01:50:17,791 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_AgentXmppConnectionStatusReq?
2026-02-09 01:50:17,799 - DEBUG - Nothing to compare xmpp stats {'10.0.0.242': {'10.0.0.129': '0'}} with
2026-02-09 01:50:17,799 - INFO - Initial checks done. Running the testcase now
2026-02-09 01:50:17,799 - INFO -
2026-02-09 01:50:17,803 - INFO - Namespace default already exists
2026-02-09 01:50:17,804 - DEBUG - [10.0.0.129]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2026-02-09 01:50:17,922 - DEBUG - Output : cluster_project={}
2026-02-09 01:50:18,029 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2026-02-09 01:50:18,029 - DEBUG - Project uuid in Contrail is 668100c7-bb7d-408d-bb3e-363efce8b4b2
2026-02-09 01:50:18,029 - DEBUG - [10.0.0.129]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2026-02-09 01:50:18,167 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2026-02-09 01:50:18,168 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_NamespaceDatabaseList?namespace_uuid=2ae2d978-4586-4420-ab4f-a379ace9383f
2026-02-09 01:50:18,172 - INFO - Namespace default with uuid 2ae2d978-4586-4420-ab4f-a379ace9383f found in kube manager
2026-02-09 01:50:18,173 - INFO - Namespace default verification passed
2026-02-09 01:50:18,175 - DEBUG - Service ctest-nginx-svc-50555479 not present
2026-02-09 01:50:18,176 - INFO - Creating service ctest-nginx-svc-50555479
2026-02-09 01:50:18,199 - INFO - Created Service ctest-nginx-svc-50555479
2026-02-09 01:50:18,208 - DEBUG - Pod ctest-nginx-pod-74147095 not present
2026-02-09 01:50:18,209 - INFO - Creating Pod ctest-nginx-pod-74147095
2026-02-09 01:50:18,253 - DEBUG - Node for Pod ctest-nginx-pod-74147095 not yet populated
2026-02-09 01:50:21,259 - DEBUG - Pod : ctest-nginx-pod-74147095 UUID is d439824b-4b89-4834-819f-493d0567091d
2026-02-09 01:50:21,262 - DEBUG - Pod ctest-nginx-pod-06307878 not present
2026-02-09 01:50:21,262 - INFO - Creating Pod ctest-nginx-pod-06307878
2026-02-09 01:50:21,285 - DEBUG - Pod : ctest-nginx-pod-06307878 UUID is 18d03ac7-73c9-42a6-8676-72d8e6ce12ba
2026-02-09 01:50:21,508 - INFO - Created VN __public__, UUID :ee3f1809-b18f-4b8e-99c8-883f36fb82bd
2026-02-09 01:50:21,785 - DEBUG - Requesting: http://10.0.0.129:8082/domains
2026-02-09 01:50:21,791 - DEBUG - Requesting: http://10.0.0.129:8082/domain/d1fcea34-5a20-4afb-8ddb-7e5cb5c4a914
2026-02-09 01:50:21,825 - DEBUG - Requesting: http://10.0.0.129:8082/project/668100c7-bb7d-408d-bb3e-363efce8b4b2
2026-02-09 01:50:21,903 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/ee3f1809-b18f-4b8e-99c8-883f36fb82bd
2026-02-09 01:50:21,917 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/ee3f1809-b18f-4b8e-99c8-883f36fb82bd
2026-02-09 01:50:21,924 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/0f8155f2-f2f6-4c0d-88a5-dc68ed5b421d
2026-02-09 01:50:21,931 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/0f8155f2-f2f6-4c0d-88a5-dc68ed5b421d
2026-02-09 01:50:21,936 - DEBUG - Requesting: http://10.0.0.129:8082/route-target/5ebcdfde-b1f8-464a-9c90-6de84da3407f
2026-02-09 01:50:21,951 - DEBUG - Route Targets: ['target:64512:8000006']
2026-02-09 01:50:21,951 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/ee3f1809-b18f-4b8e-99c8-883f36fb82bd
2026-02-09 01:50:21,969 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/0f8155f2-f2f6-4c0d-88a5-dc68ed5b421d
2026-02-09 01:50:21,981 - INFO - Verified VN network id 13 for VN ee3f1809-b18f-4b8e-99c8-883f36fb82bd
2026-02-09 01:50:21,981 - INFO - Verifications in API Server for VN __public__ passed
2026-02-09 01:50:21,981 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/ee3f1809-b18f-4b8e-99c8-883f36fb82bd
2026-02-09 01:50:21,989 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/0f8155f2-f2f6-4c0d-88a5-dc68ed5b421d
2026-02-09 01:50:21,995 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/0f8155f2-f2f6-4c0d-88a5-dc68ed5b421d
2026-02-09 01:50:22,001 - DEBUG - Requesting: http://10.0.0.129:8082/route-target/5ebcdfde-b1f8-464a-9c90-6de84da3407f
2026-02-09 01:50:22,007 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2026-02-09 01:50:22,014 - DEBUG - Control-node 10.0.0.129 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17167466734472350606', 'uuid-lslong': '11081256688231875261'}, 'enable': 'true', 'created': '2026-02-09T01:50:21', 'last-modified': '2026-02-09T01:50:21', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.363030'}
2026-02-09 01:50:22,014 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2026-02-09 01:50:22,025 - DEBUG - Route Targets: ['target:64512:8000006']
2026-02-09 01:50:22,025 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2026-02-09 01:50:22,026 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2026-02-09 01:50:22,027 - DEBUG - Requesting: http://10.0.0.129:8082/domains
2026-02-09 01:50:22,035 - DEBUG - Requesting: http://10.0.0.129:8082/domain/d1fcea34-5a20-4afb-8ddb-7e5cb5c4a914
2026-02-09 01:50:22,044 - DEBUG - Requesting: http://10.0.0.129:8082/project/668100c7-bb7d-408d-bb3e-363efce8b4b2
2026-02-09 01:50:22,091 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/ee3f1809-b18f-4b8e-99c8-883f36fb82bd
2026-02-09 01:50:22,100 - DEBUG - =>VN __public__ has no policy to be verified
2026-02-09 01:50:22,100 - DEBUG - Verifying the vn in opserver
2026-02-09 01:50:22,100 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.129
2026-02-09 01:50:22,100 - DEBUG - Requesting: http://10.0.0.129:8081/analytics/uves/virtual-networks
2026-02-09 01:50:22,105 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.129:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2026-02-09 01:50:22,105 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2026-02-09 01:50:22,105 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-02-09 01:50:22,115 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-02-09 01:50:22,127 - DEBUG - Do not have enough data to verify VN in agent
2026-02-09 01:50:22,127 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-02-09 01:50:22,139 - DEBUG - VRF ids for VN __public__: {}
2026-02-09 01:50:22,239 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2026-02-09 01:50:22,347 - DEBUG - Requesting: http://10.0.0.129:8082/domains
2026-02-09 01:50:22,352 - DEBUG - Requesting: http://10.0.0.129:8082/domain/d1fcea34-5a20-4afb-8ddb-7e5cb5c4a914
2026-02-09 01:50:22,358 - DEBUG - Requesting: http://10.0.0.129:8082/project/668100c7-bb7d-408d-bb3e-363efce8b4b2
2026-02-09 01:50:22,394 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/ee3f1809-b18f-4b8e-99c8-883f36fb82bd
2026-02-09 01:50:22,404 - DEBUG - Requesting: http://10.0.0.129:8082/floating-ip-pool/c1508870-5fee-43b4-904f-3cbda14a90ca
2026-02-09 01:50:22,410 - DEBUG - Requesting: http://10.0.0.129:8082/domains
2026-02-09 01:50:22,417 - DEBUG - Requesting: http://10.0.0.129:8082/domain/d1fcea34-5a20-4afb-8ddb-7e5cb5c4a914
2026-02-09 01:50:22,424 - DEBUG - Requesting: http://10.0.0.129:8082/project/668100c7-bb7d-408d-bb3e-363efce8b4b2
2026-02-09 01:50:22,471 - DEBUG - Requesting: http://10.0.0.129:8082/virtual-network/ee3f1809-b18f-4b8e-99c8-883f36fb82bd
2026-02-09 01:50:22,481 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2026-02-09 01:50:22,481 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2026-02-09 01:50:22,491 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2026-02-09 01:50:22,499 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2026-02-09 01:50:22,499 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2026-02-09 01:50:22,502 - DEBUG - Ingress ctest-nginx-ingress-45075752 not present
2026-02-09 01:50:22,503 - INFO - Creating Ingress ctest-nginx-ingress-45075752
2026-02-09 01:50:22,516 - INFO - Created Ingress ctest-nginx-ingress-45075752
2026-02-09 01:50:22,522 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-45075752
2026-02-09 01:50:25,528 - DEBUG - For Ingress ctest-nginx-ingress-45075752, Cluster IP: 10.47.255.249, External IPs ['77.116.16.60']
2026-02-09 01:50:25,529 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-45075752
2026-02-09 01:50:25,529 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_LoadbalancerDatabaseList?x=47ab3ebe-ff1e-442e-9602-c7992f0944a7
2026-02-09 01:50:25,534 - INFO - Ingress ctest-nginx-ingress-45075752 with uuid 47ab3ebe-ff1e-442e-9602-c7992f0944a7 found in kube manager
2026-02-09 01:50:25,534 - INFO - Ingress ctest-nginx-ingress-45075752 verification passed
2026-02-09 01:50:25,537 - DEBUG - Pod ctest-busybox-pod-51643080 not present
2026-02-09 01:50:25,538 - INFO - Creating Pod ctest-busybox-pod-51643080
2026-02-09 01:50:25,568 - DEBUG - Pod : ctest-busybox-pod-51643080 UUID is 507bc8ae-1a8e-4c38-b296-403448e69970
2026-02-09 01:50:25,575 - DEBUG - Pod ctest-nginx-pod-74147095 not in running state.Currently in Pending
2026-02-09 01:50:30,586 - INFO - Pod ctest-nginx-pod-74147095 is in running state.Got IP 10.47.255.251
2026-02-09 01:50:30,603 - DEBUG - Pod ctest-nginx-pod-74147095 has vmi b07c08ba-0559-11f1-8b80-fa163e55a104
2026-02-09 01:50:30,603 - INFO - Verified pod ctest-nginx-pod-74147095 in contrail-api
2026-02-09 01:50:30,603 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:50:30,618 - DEBUG - VMI b07c08ba-0559-11f1-8b80-fa163e55a104 is active in agent 10.0.0.242
2026-02-09 01:50:30,618 - INFO - Verified Pod ctest-nginx-pod-74147095 in agent 10.0.0.242
2026-02-09 01:50:30,619 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_VirtualMachineDatabaseList?x=d439824b-4b89-4834-819f-493d0567091d
2026-02-09 01:50:30,624 - INFO - Pod ctest-nginx-pod-74147095 with uuid d439824b-4b89-4834-819f-493d0567091d found in kube manager
2026-02-09 01:50:30,624 - INFO - Pod ctest-nginx-pod-74147095 verification passed
2026-02-09 01:50:30,696 - DEBUG - [Pod ctest-nginx-pod-74147095] Cmd: echo ctest-nginx-pod-74147095 > /usr/share/nginx/html/index.html, Output:
2026-02-09 01:50:30,705 - DEBUG - Pod ctest-nginx-pod-06307878 not in running state.Currently in Pending
2026-02-09 01:50:35,717 - INFO - Pod ctest-nginx-pod-06307878 is in running state.Got IP 10.47.255.250
2026-02-09 01:50:35,773 - DEBUG - Pod ctest-nginx-pod-06307878 has vmi b190a60c-0559-11f1-8b80-fa163e55a104
2026-02-09 01:50:35,773 - INFO - Verified pod ctest-nginx-pod-06307878 in contrail-api
2026-02-09 01:50:35,773 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:50:35,784 - DEBUG - VMI b190a60c-0559-11f1-8b80-fa163e55a104 is active in agent 10.0.0.242
2026-02-09 01:50:35,784 - INFO - Verified Pod ctest-nginx-pod-06307878 in agent 10.0.0.242
2026-02-09 01:50:35,784 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_VirtualMachineDatabaseList?x=18d03ac7-73c9-42a6-8676-72d8e6ce12ba
2026-02-09 01:50:35,788 - INFO - Pod ctest-nginx-pod-06307878 with uuid 18d03ac7-73c9-42a6-8676-72d8e6ce12ba found in kube manager
2026-02-09 01:50:35,788 - INFO - Pod ctest-nginx-pod-06307878 verification passed
2026-02-09 01:50:35,841 - DEBUG - [Pod ctest-nginx-pod-06307878] Cmd: echo ctest-nginx-pod-06307878 > /usr/share/nginx/html/index.html, Output:
2026-02-09 01:50:35,848 - DEBUG - Pod ctest-busybox-pod-51643080 not in running state.Currently in Pending
2026-02-09 01:50:40,857 - INFO - Pod ctest-busybox-pod-51643080 is in running state.Got IP 10.47.255.248
2026-02-09 01:50:40,871 - DEBUG - Pod ctest-busybox-pod-51643080 has vmi b42a2802-0559-11f1-8b80-fa163e55a104
2026-02-09 01:50:40,871 - INFO - Verified pod ctest-busybox-pod-51643080 in contrail-api
2026-02-09 01:50:40,871 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:50:40,883 - DEBUG - VMI b42a2802-0559-11f1-8b80-fa163e55a104 is active in agent 10.0.0.242
2026-02-09 01:50:40,883 - INFO - Verified Pod ctest-busybox-pod-51643080 in agent 10.0.0.242
2026-02-09 01:50:40,883 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_VirtualMachineDatabaseList?x=507bc8ae-1a8e-4c38-b296-403448e69970
2026-02-09 01:50:40,888 - INFO - Pod ctest-busybox-pod-51643080 with uuid 507bc8ae-1a8e-4c38-b296-403448e69970 found in kube manager
2026-02-09 01:50:40,888 - INFO - Pod ctest-busybox-pod-51643080 verification passed
2026-02-09 01:50:40,943 - DEBUG - [Pod ctest-busybox-pod-51643080] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-06307878
2026-02-09 01:50:40,943 - DEBUG - [Pod Pod ctest-busybox-pod-51643080] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2026-02-09 01:50:40,943 - DEBUG - [Pod Pod ctest-busybox-pod-51643080] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-06307878
2026-02-09 01:50:41,005 - DEBUG - [Pod ctest-busybox-pod-51643080] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-74147095
2026-02-09 01:50:41,006 - DEBUG - [Pod Pod ctest-busybox-pod-51643080] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2026-02-09 01:50:41,006 - DEBUG - [Pod Pod ctest-busybox-pod-51643080] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-74147095
2026-02-09 01:50:41,006 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-74147095': 1, 'ctest-nginx-pod-06307878': 1}
2026-02-09 01:50:41,006 - WARNING - Nothing to delete parallely
2026-02-09 01:50:41,007 - INFO - Deleting pod default:ctest-busybox-pod-51643080
2026-02-09 01:50:41,022 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=507bc8ae-1a8e-4c38-b296-403448e69970
2026-02-09 01:50:41,029 - WARNING - Pod uuid 507bc8ae-1a8e-4c38-b296-403448e69970 is still seen in agent 10.0.0.242 VM list
2026-02-09 01:50:46,030 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=507bc8ae-1a8e-4c38-b296-403448e69970
2026-02-09 01:50:46,036 - WARNING - Pod uuid 507bc8ae-1a8e-4c38-b296-403448e69970 is still seen in agent 10.0.0.242 VM list
2026-02-09 01:50:51,037 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=507bc8ae-1a8e-4c38-b296-403448e69970
2026-02-09 01:50:51,043 - WARNING - Pod uuid 507bc8ae-1a8e-4c38-b296-403448e69970 is still seen in agent 10.0.0.242 VM list
2026-02-09 01:50:56,045 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=507bc8ae-1a8e-4c38-b296-403448e69970
2026-02-09 01:50:56,052 - WARNING - Pod uuid 507bc8ae-1a8e-4c38-b296-403448e69970 is still seen in agent 10.0.0.242 VM list
2026-02-09 01:51:01,053 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=507bc8ae-1a8e-4c38-b296-403448e69970
2026-02-09 01:51:01,061 - WARNING - Pod uuid 507bc8ae-1a8e-4c38-b296-403448e69970 is still seen in agent 10.0.0.242 VM list
2026-02-09 01:51:06,061 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=507bc8ae-1a8e-4c38-b296-403448e69970
2026-02-09 01:51:06,069 - WARNING - Pod uuid 507bc8ae-1a8e-4c38-b296-403448e69970 is still seen in agent 10.0.0.242 VM list
2026-02-09 01:51:11,070 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=507bc8ae-1a8e-4c38-b296-403448e69970
2026-02-09 01:51:11,076 - WARNING - Pod uuid 507bc8ae-1a8e-4c38-b296-403448e69970 is still seen in agent 10.0.0.242 VM list
2026-02-09 01:51:16,077 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=507bc8ae-1a8e-4c38-b296-403448e69970
2026-02-09 01:51:16,089 - DEBUG - Pod 507bc8ae-1a8e-4c38-b296-403448e69970 is not in agent 10.0.0.242 VM list
2026-02-09 01:51:16,089 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b42a2802-0559-11f1-8b80-fa163e55a104
2026-02-09 01:51:16,097 - DEBUG - VMI b42a2802-0559-11f1-8b80-fa163e55a104 is removed from agent 10.0.0.242
2026-02-09 01:51:16,097 - INFO - Verified that pod ctest-busybox-pod-51643080 is removed in agent
2026-02-09 01:51:16,097 - INFO - Deleting Ingress : ctest-nginx-ingress-45075752
2026-02-09 01:51:16,104 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_LoadbalancerDatabaseList?x=47ab3ebe-ff1e-442e-9602-c7992f0944a7
2026-02-09 01:51:16,110 - ERROR - Ingress ctest-nginx-ingress-45075752 with uuid 47ab3ebe-ff1e-442e-9602-c7992f0944a7 still found in kube manager
2026-02-09 01:51:17,110 - DEBUG - Requesting: http://10.0.0.129:8108/Snh_LoadbalancerDatabaseList?x=47ab3ebe-ff1e-442e-9602-c7992f0944a7
2026-02-09 01:51:17,121 - INFO - Ingress ctest-nginx-ingress-45075752 with uuid 47ab3ebe-ff1e-442e-9602-c7992f0944a7 deleted successfully from kube manager
2026-02-09 01:51:17,123 - INFO - Verifications on Ingress %s cleanup passed
2026-02-09 01:51:17,123 - INFO - Deleting the FIP pool __fip_pool_public__
2026-02-09 01:51:17,304 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2026-02-09 01:51:17,314 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2026-02-09 01:51:17,314 - INFO - Deleting VN __public__
2026-02-09 01:51:17,867 - DEBUG - Requesting: http://10.0.0.129:8082/routing-instance/0f8155f2-f2f6-4c0d-88a5-dc68ed5b421d
2026-02-09 01:51:17,875 - DEBUG - Response Code: 404
2026-02-09 01:51:17,876 - DEBUG - Requesting: http://10.0.0.129:8082/domains
2026-02-09 01:51:17,890 - DEBUG - Requesting: http://10.0.0.129:8082/domain/d1fcea34-5a20-4afb-8ddb-7e5cb5c4a914
2026-02-09 01:51:17,941 - DEBUG - Requesting: http://10.0.0.129:8082/project/668100c7-bb7d-408d-bb3e-363efce8b4b2
2026-02-09 01:51:18,034 - INFO - Validated that VN __public__ is not found in API Server
2026-02-09 01:51:18,034 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2026-02-09 01:51:18,047 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-02-09 01:51:18,073 - DEBUG - VN __public__ is not present in Agent 10.0.0.242
2026-02-09 01:51:18,074 - INFO - Validated that VN __public__ is not in any agent
2026-02-09 01:51:18,074 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2026-02-09 01:51:18,085 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2026-02-09 01:51:18,106 - DEBUG - Requesting: http://10.0.0.129:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2026-02-09 01:51:18,133 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2026-02-09 01:51:18,134 - INFO - Deleting pod default:ctest-nginx-pod-06307878
2026-02-09 01:51:18,184 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=18d03ac7-73c9-42a6-8676-72d8e6ce12ba
2026-02-09 01:51:18,193 - WARNING - Pod uuid 18d03ac7-73c9-42a6-8676-72d8e6ce12ba is still seen in agent 10.0.0.242 VM list
2026-02-09 01:51:23,197 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=18d03ac7-73c9-42a6-8676-72d8e6ce12ba
2026-02-09 01:51:23,204 - DEBUG - Pod 18d03ac7-73c9-42a6-8676-72d8e6ce12ba is not in agent 10.0.0.242 VM list
2026-02-09 01:51:23,204 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b190a60c-0559-11f1-8b80-fa163e55a104
2026-02-09 01:51:23,210 - DEBUG - VMI b190a60c-0559-11f1-8b80-fa163e55a104 is removed from agent 10.0.0.242
2026-02-09 01:51:23,210 - INFO - Verified that pod ctest-nginx-pod-06307878 is removed in agent
2026-02-09 01:51:23,210 - INFO - Deleting pod default:ctest-nginx-pod-74147095
2026-02-09 01:51:23,223 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=d439824b-4b89-4834-819f-493d0567091d
2026-02-09 01:51:23,231 - WARNING - Pod uuid d439824b-4b89-4834-819f-493d0567091d is still seen in agent 10.0.0.242 VM list
2026-02-09 01:51:28,232 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=d439824b-4b89-4834-819f-493d0567091d
2026-02-09 01:51:28,251 - DEBUG - Pod d439824b-4b89-4834-819f-493d0567091d is not in agent 10.0.0.242 VM list
2026-02-09 01:51:28,251 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b07c08ba-0559-11f1-8b80-fa163e55a104
2026-02-09 01:51:28,267 - DEBUG - VMI b07c08ba-0559-11f1-8b80-fa163e55a104 is removed from agent 10.0.0.242
2026-02-09 01:51:28,267 - INFO - Verified that pod ctest-nginx-pod-74147095 is removed in agent
2026-02-09 01:51:28,267 - INFO - Deleting service : ctest-nginx-svc-50555479
2026-02-09 01:51:28,465 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_AgentXmppConnectionStatusReq?
2026-02-09 01:51:28,472 - DEBUG - No XMPP flaps were noticed during the test
2026-02-09 01:51:28,472 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:11]
2026-02-09 01:51:28,472 - INFO - --------------------------------------------------------------------------------
2026-02-09 01:51:35,016 - DEBUG - [10.0.0.129]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2026-02-09 01:51:35,212 - DEBUG - Output : cluster_project={}
2026-02-09 01:51:35,251 - DEBUG - [10.0.0.129]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2026-02-09 01:51:35,474 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2026-02-09 01:51:35,482 - DEBUG - [10.0.0.129]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2026-02-09 01:51:35,625 - DEBUG - Output : cluster_project={}
2026-02-09 01:51:50,777 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:51:50,813 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:51:50,888 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:51:50,969 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:51:51,004 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:51:56,093 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:51:56,139 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:51:56,220 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2026-02-09 01:51:56,443 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_AgentXmppConnectionStatusReq?
2026-02-09 01:51:59,046 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_ApplicationPolicySetReq?
2026-02-09 01:51:59,053 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_AclReq?
2026-02-09 01:51:59,077 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_AclReq?
2026-02-09 01:52:01,101 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_AclReq?
2026-02-09 01:55:09,871 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_AgentXmppConnectionStatusReq?
2026-02-09 01:55:09,919 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=2a4c632c-436d-4c75-b05d-821d0d8de30c
2026-02-09 01:55:09,937 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=76380289-fc1c-462a-8f96-9c3838b795d1
2026-02-09 01:55:09,959 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=3ce402d2-50df-4d00-b13f-a38984ecbf40
2026-02-09 01:55:09,966 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=fd7908db-978d-43d4-86f3-f379eff1f93c
2026-02-09 01:55:09,993 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=16a0024a-0586-4441-aa22-a05f34368b8b
2026-02-09 01:55:10,006 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=0042a413-3628-42a6-b313-153604924ec7
2026-02-09 01:55:10,010 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=7422a686-f218-4e2e-88a3-7567a26b5c8e
2026-02-09 01:55:10,016 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=e0a92119-b25b-4d12-91f1-e827c68cd953
2026-02-09 01:55:15,018 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=2a4c632c-436d-4c75-b05d-821d0d8de30c
2026-02-09 01:55:15,023 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=fd7908db-978d-43d4-86f3-f379eff1f93c
2026-02-09 01:55:15,026 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:de502faa-0559-11f1-8b80-fa163e55a104
2026-02-09 01:55:15,028 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=3ce402d2-50df-4d00-b13f-a38984ecbf40
2026-02-09 01:55:15,029 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=76380289-fc1c-462a-8f96-9c3838b795d1
2026-02-09 01:55:15,031 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=0042a413-3628-42a6-b313-153604924ec7
2026-02-09 01:55:15,033 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=16a0024a-0586-4441-aa22-a05f34368b8b
2026-02-09 01:55:15,034 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=e0a92119-b25b-4d12-91f1-e827c68cd953
2026-02-09 01:55:15,035 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=7422a686-f218-4e2e-88a3-7567a26b5c8e
2026-02-09 01:55:15,049 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:df6e2dc4-0559-11f1-8b80-fa163e55a104
2026-02-09 01:55:15,054 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e0004cb8-0559-11f1-8b80-fa163e55a104
2026-02-09 01:55:15,055 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:decf67fc-0559-11f1-8b80-fa163e55a104
2026-02-09 01:55:20,049 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=0042a413-3628-42a6-b313-153604924ec7
2026-02-09 01:55:20,054 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=e0a92119-b25b-4d12-91f1-e827c68cd953
2026-02-09 01:55:20,055 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=7422a686-f218-4e2e-88a3-7567a26b5c8e
2026-02-09 01:55:20,056 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=76380289-fc1c-462a-8f96-9c3838b795d1
2026-02-09 01:55:25,059 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=0042a413-3628-42a6-b313-153604924ec7
2026-02-09 01:55:25,066 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=e0a92119-b25b-4d12-91f1-e827c68cd953
2026-02-09 01:55:25,068 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=76380289-fc1c-462a-8f96-9c3838b795d1
2026-02-09 01:55:25,068 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=7422a686-f218-4e2e-88a3-7567a26b5c8e
2026-02-09 01:55:30,068 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=0042a413-3628-42a6-b313-153604924ec7
2026-02-09 01:55:30,078 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=e0a92119-b25b-4d12-91f1-e827c68cd953
2026-02-09 01:55:30,079 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=7422a686-f218-4e2e-88a3-7567a26b5c8e
2026-02-09 01:55:30,080 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=76380289-fc1c-462a-8f96-9c3838b795d1
2026-02-09 01:55:35,078 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=0042a413-3628-42a6-b313-153604924ec7
2026-02-09 01:55:35,093 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=e0a92119-b25b-4d12-91f1-e827c68cd953
2026-02-09 01:55:35,094 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=7422a686-f218-4e2e-88a3-7567a26b5c8e
2026-02-09 01:55:35,096 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=76380289-fc1c-462a-8f96-9c3838b795d1
2026-02-09 01:55:40,088 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=0042a413-3628-42a6-b313-153604924ec7
2026-02-09 01:55:40,107 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=7422a686-f218-4e2e-88a3-7567a26b5c8e
2026-02-09 01:55:40,108 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=76380289-fc1c-462a-8f96-9c3838b795d1
2026-02-09 01:55:40,109 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=e0a92119-b25b-4d12-91f1-e827c68cd953
2026-02-09 01:55:45,131 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=7422a686-f218-4e2e-88a3-7567a26b5c8e
2026-02-09 01:55:45,133 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=76380289-fc1c-462a-8f96-9c3838b795d1
2026-02-09 01:55:45,134 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=e0a92119-b25b-4d12-91f1-e827c68cd953
2026-02-09 01:55:45,134 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_VmListReq?uuid=0042a413-3628-42a6-b313-153604924ec7
2026-02-09 01:55:45,145 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e08cec86-0559-11f1-8b80-fa163e55a104
2026-02-09 01:55:45,147 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e102c87a-0559-11f1-8b80-fa163e55a104
2026-02-09 01:55:45,148 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e1782f84-0559-11f1-8b80-fa163e55a104
2026-02-09 01:55:45,149 - DEBUG - Requesting: http://10.0.0.242:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e1e53778-0559-11f1-8b80-fa163e55a104