2025-03-13 02:12:16,155 - DEBUG - [10.0.0.12]: Running cmd : hostname
2025-03-13 02:12:16,408 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1009-1
2025-03-13 02:12:16,408 - DEBUG - [10.0.0.12]: Running cmd : hostname -f
2025-03-13 02:12:16,452 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1009-1
2025-03-13 02:12:16,452 - DEBUG - [10.0.0.12]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-03-13 02:12:16,557 - DEBUG - Output : NAMES
contrail_test_tCh3iMMa6
contrail-kubernetes-master_kubemanager_1
analytics_database_query-engine_1
analytics_database_cassandra_1
analytics_database_nodemgr_1
analytics_snmp_topology_1
analytics_snmp_nodemgr_1
analytics_snmp_snmp-collector_1
analytics_alarm_kafka_1
analytics_alarm_nodemgr_1
analytics_alarm_alarm-gen_1
analytics_nodemgr_1
analytics_collector_1
analytics_api_1
redis_redis_1
webui_job_1
webui_web_1
control_named_1
control_nodemgr_1
control_dns_1
control_control_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_rabbitmq_1
config_database_cassandra_1
config_api_devicemgr_1
config_api_schema_1
config_api_svcmonitor_1
config_api_dnsmasq_1
config_api_nodemgr_1
config_api_api_1
2025-03-13 02:12:16,558 - DEBUG - [10.0.0.12]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-03-13 02:12:16,631 - DEBUG - Output : 10.0.0.12/24
172.17.0.1/16
2025-03-13 02:12:16,631 - DEBUG - [10.0.0.12]: Running cmd : getent hosts 10.0.0.12 | head -n 1 | awk '{print $2}'
2025-03-13 02:12:16,664 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1009-1
2025-03-13 02:12:16,732 - DEBUG - [10.0.0.28]: Running cmd : hostname
2025-03-13 02:12:16,942 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1009-1
2025-03-13 02:12:16,943 - DEBUG - [10.0.0.28]: Running cmd : hostname -f
2025-03-13 02:12:16,968 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1009-1
2025-03-13 02:12:16,969 - DEBUG - [10.0.0.28]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-03-13 02:12:17,015 - DEBUG - Output : NAMES
vrouter_nodemgr_1
vrouter_vrouter-agent_1
2025-03-13 02:12:17,015 - DEBUG - [10.0.0.28]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-03-13 02:12:17,089 - DEBUG - Output : 10.0.0.28/24
2025-03-13 02:12:17,090 - DEBUG - [10.0.0.28]: Running cmd : getent hosts 10.0.0.28 | head -n 1 | awk '{print $2}'
2025-03-13 02:12:17,125 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1009-1
2025-03-13 02:12:17,210 - INFO - Using existing project ['default-domain', 'k8s-default'](b4ef722d-dac4-45a8-a477-60403d029a2d)
2025-03-13 02:12:17,361 - INFO - Using existing project ['default-domain', 'k8s-default'](b4ef722d-dac4-45a8-a477-60403d029a2d)
2025-03-13 02:12:17,543 - INFO - ================================================================================
2025-03-13 02:12:17,543 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-03-13 02:12:17,544 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-03-13 02:12:17,799 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_AgentXmppConnectionStatusReq?
2025-03-13 02:12:17,809 - DEBUG - Nothing to compare xmpp stats {'10.0.0.28': {'10.0.0.12': '0'}} with
2025-03-13 02:12:17,809 - INFO - Initial checks done. Running the testcase now
2025-03-13 02:12:17,809 - INFO -
2025-03-13 02:12:17,814 - INFO - Namespace default already exists
2025-03-13 02:12:17,815 - DEBUG - [10.0.0.12]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-03-13 02:12:17,977 - DEBUG - Output : cluster_project={}
2025-03-13 02:12:18,035 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-03-13 02:12:18,036 - DEBUG - Project uuid in Contrail is b4ef722d-dac4-45a8-a477-60403d029a2d
2025-03-13 02:12:18,036 - DEBUG - [10.0.0.12]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-03-13 02:12:18,182 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-03-13 02:12:18,183 - DEBUG - Requesting: http://10.0.0.12:8108/Snh_NamespaceDatabaseList?namespace_uuid=10b01592-9b67-4887-92b0-8cec817dc69f
2025-03-13 02:12:18,188 - INFO - Namespace default with uuid 10b01592-9b67-4887-92b0-8cec817dc69f found in kube manager
2025-03-13 02:12:18,188 - INFO - Namespace default verification passed
2025-03-13 02:12:18,198 - DEBUG - Service ctest-nginx-svc-13093608 not present
2025-03-13 02:12:18,198 - INFO - Creating service ctest-nginx-svc-13093608
2025-03-13 02:12:18,219 - INFO - Created Service ctest-nginx-svc-13093608
2025-03-13 02:12:18,225 - DEBUG - Pod ctest-nginx-pod-99249441 not present
2025-03-13 02:12:18,225 - INFO - Creating Pod ctest-nginx-pod-99249441
2025-03-13 02:12:18,272 - DEBUG - Pod : ctest-nginx-pod-99249441 UUID is 478dfad1-c620-4736-b979-dfb4cde254db
2025-03-13 02:12:18,277 - DEBUG - Pod ctest-nginx-pod-72312214 not present
2025-03-13 02:12:18,277 - INFO - Creating Pod ctest-nginx-pod-72312214
2025-03-13 02:12:18,328 - DEBUG - Node for Pod ctest-nginx-pod-72312214 not yet populated
2025-03-13 02:12:21,337 - DEBUG - Pod : ctest-nginx-pod-72312214 UUID is 34b36a5c-aad1-4fb0-a099-e66776188fc7
2025-03-13 02:12:21,587 - INFO - Created VN __public__, UUID :cac6f10f-3fe1-4d32-be33-1718854e3431
2025-03-13 02:12:21,808 - DEBUG - Requesting: http://10.0.0.12:8082/domains
2025-03-13 02:12:21,818 - DEBUG - Requesting: http://10.0.0.12:8082/domain/54cacbf0-0712-46f8-9c8a-7546eb095f7b
2025-03-13 02:12:21,846 - DEBUG - Requesting: http://10.0.0.12:8082/project/b4ef722d-dac4-45a8-a477-60403d029a2d
2025-03-13 02:12:21,910 - DEBUG - Requesting: http://10.0.0.12:8082/virtual-network/cac6f10f-3fe1-4d32-be33-1718854e3431
2025-03-13 02:12:21,929 - DEBUG - Requesting: http://10.0.0.12:8082/virtual-network/cac6f10f-3fe1-4d32-be33-1718854e3431
2025-03-13 02:12:21,941 - DEBUG - Requesting: http://10.0.0.12:8082/routing-instance/c279f83c-38a8-4daf-9cb8-cf5eebde000d
2025-03-13 02:12:21,950 - DEBUG - Requesting: http://10.0.0.12:8082/routing-instance/c279f83c-38a8-4daf-9cb8-cf5eebde000d
2025-03-13 02:12:21,959 - DEBUG - Requesting: http://10.0.0.12:8082/route-target/8b3e9b79-cfd9-4baa-ab78-e9e6d907bcdd
2025-03-13 02:12:21,969 - DEBUG - Route Targets: ['target:64512:8000006']
2025-03-13 02:12:21,969 - DEBUG - Requesting: http://10.0.0.12:8082/virtual-network/cac6f10f-3fe1-4d32-be33-1718854e3431
2025-03-13 02:12:21,982 - DEBUG - Requesting: http://10.0.0.12:8082/routing-instance/c279f83c-38a8-4daf-9cb8-cf5eebde000d
2025-03-13 02:12:22,004 - INFO - Verified VN network id 13 for VN cac6f10f-3fe1-4d32-be33-1718854e3431
2025-03-13 02:12:22,005 - INFO - Verifications in API Server for VN __public__ passed
2025-03-13 02:12:22,005 - DEBUG - Requesting: http://10.0.0.12:8082/virtual-network/cac6f10f-3fe1-4d32-be33-1718854e3431
2025-03-13 02:12:22,019 - DEBUG - Requesting: http://10.0.0.12:8082/routing-instance/c279f83c-38a8-4daf-9cb8-cf5eebde000d
2025-03-13 02:12:22,028 - DEBUG - Requesting: http://10.0.0.12:8082/routing-instance/c279f83c-38a8-4daf-9cb8-cf5eebde000d
2025-03-13 02:12:22,039 - DEBUG - Requesting: http://10.0.0.12:8082/route-target/8b3e9b79-cfd9-4baa-ab78-e9e6d907bcdd
2025-03-13 02:12:22,062 - DEBUG - Requesting: http://10.0.0.12:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-03-13 02:12:22,070 - DEBUG - Control-node 10.0.0.12 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14611631088848686386', 'uuid-lslong': '13705323485101700145'}, 'enable': 'true', 'created': '2025-03-13T02:12:21', 'last-modified': '2025-03-13T02:12:21', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.313348'}
2025-03-13 02:12:22,070 - DEBUG - Requesting: http://10.0.0.12:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-03-13 02:12:22,079 - DEBUG - Route Targets: ['target:64512:8000006']
2025-03-13 02:12:22,079 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-03-13 02:12:22,080 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-03-13 02:12:22,080 - DEBUG - Requesting: http://10.0.0.12:8082/domains
2025-03-13 02:12:22,088 - DEBUG - Requesting: http://10.0.0.12:8082/domain/54cacbf0-0712-46f8-9c8a-7546eb095f7b
2025-03-13 02:12:22,102 - DEBUG - Requesting: http://10.0.0.12:8082/project/b4ef722d-dac4-45a8-a477-60403d029a2d
2025-03-13 02:12:22,169 - DEBUG - Requesting: http://10.0.0.12:8082/virtual-network/cac6f10f-3fe1-4d32-be33-1718854e3431
2025-03-13 02:12:22,189 - DEBUG - =>VN __public__ has no policy to be verified
2025-03-13 02:12:22,189 - DEBUG - Verifying the vn in opserver
2025-03-13 02:12:22,189 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.12
2025-03-13 02:12:22,189 - DEBUG - Requesting: http://10.0.0.12:8081/analytics/uves/virtual-networks
2025-03-13 02:12:22,196 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.12:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-03-13 02:12:22,196 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-03-13 02:12:22,197 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-03-13 02:12:22,206 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-03-13 02:12:22,213 - DEBUG - Do not have enough data to verify VN in agent
2025-03-13 02:12:22,214 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-03-13 02:12:22,221 - DEBUG - VRF ids for VN __public__: {}
2025-03-13 02:12:22,332 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-03-13 02:12:22,460 - DEBUG - Requesting: http://10.0.0.12:8082/domains
2025-03-13 02:12:22,469 - DEBUG - Requesting: http://10.0.0.12:8082/domain/54cacbf0-0712-46f8-9c8a-7546eb095f7b
2025-03-13 02:12:22,478 - DEBUG - Requesting: http://10.0.0.12:8082/project/b4ef722d-dac4-45a8-a477-60403d029a2d
2025-03-13 02:12:22,534 - DEBUG - Requesting: http://10.0.0.12:8082/virtual-network/cac6f10f-3fe1-4d32-be33-1718854e3431
2025-03-13 02:12:22,551 - DEBUG - Requesting: http://10.0.0.12:8082/floating-ip-pool/85a5188c-0b04-4cfb-94d0-44528159b711
2025-03-13 02:12:22,563 - DEBUG - Requesting: http://10.0.0.12:8082/domains
2025-03-13 02:12:22,571 - DEBUG - Requesting: http://10.0.0.12:8082/domain/54cacbf0-0712-46f8-9c8a-7546eb095f7b
2025-03-13 02:12:22,589 - DEBUG - Requesting: http://10.0.0.12:8082/project/b4ef722d-dac4-45a8-a477-60403d029a2d
2025-03-13 02:12:22,653 - DEBUG - Requesting: http://10.0.0.12:8082/virtual-network/cac6f10f-3fe1-4d32-be33-1718854e3431
2025-03-13 02:12:22,666 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-03-13 02:12:22,666 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-03-13 02:12:22,686 - DEBUG - Requesting: http://10.0.0.12:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-03-13 02:12:22,695 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-03-13 02:12:22,696 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-03-13 02:12:22,698 - DEBUG - Ingress ctest-nginx-ingress-32063800 not present
2025-03-13 02:12:22,699 - INFO - Creating Ingress ctest-nginx-ingress-32063800
2025-03-13 02:12:22,708 - INFO - Created Ingress ctest-nginx-ingress-32063800
2025-03-13 02:12:22,714 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-32063800
2025-03-13 02:12:25,719 - DEBUG - For Ingress ctest-nginx-ingress-32063800, Cluster IP: 10.47.255.249, External IPs ['99.203.34.252']
2025-03-13 02:12:25,719 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-32063800
2025-03-13 02:12:25,720 - DEBUG - Requesting: http://10.0.0.12:8108/Snh_LoadbalancerDatabaseList?x=c9ef9119-b15f-4c40-b51b-66b1cbfc67a6
2025-03-13 02:12:25,727 - INFO - Ingress ctest-nginx-ingress-32063800 with uuid c9ef9119-b15f-4c40-b51b-66b1cbfc67a6 found in kube manager
2025-03-13 02:12:25,728 - INFO - Ingress ctest-nginx-ingress-32063800 verification passed
2025-03-13 02:12:25,731 - DEBUG - Pod ctest-busybox-pod-90941759 not present
2025-03-13 02:12:25,731 - INFO - Creating Pod ctest-busybox-pod-90941759
2025-03-13 02:12:25,771 - DEBUG - Pod : ctest-busybox-pod-90941759 UUID is 494bbf83-9a46-4234-bef9-30b43736456e
2025-03-13 02:12:25,781 - DEBUG - Pod ctest-nginx-pod-99249441 not in running state.Currently in Pending
2025-03-13 02:12:30,788 - INFO - Pod ctest-nginx-pod-99249441 is in running state.Got IP 10.47.255.251
2025-03-13 02:12:30,849 - DEBUG - Pod ctest-nginx-pod-99249441 has vmi 97fa5e40-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:12:30,849 - INFO - Verified pod ctest-nginx-pod-99249441 in contrail-api
2025-03-13 02:12:30,850 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:12:30,865 - DEBUG - VMI 97fa5e40-ffb0-11ef-af8a-fa163e56332f is active in agent 10.0.0.28
2025-03-13 02:12:30,865 - INFO - Verified Pod ctest-nginx-pod-99249441 in agent 10.0.0.28
2025-03-13 02:12:30,865 - DEBUG - Requesting: http://10.0.0.12:8108/Snh_VirtualMachineDatabaseList?x=478dfad1-c620-4736-b979-dfb4cde254db
2025-03-13 02:12:30,871 - INFO - Pod ctest-nginx-pod-99249441 with uuid 478dfad1-c620-4736-b979-dfb4cde254db found in kube manager
2025-03-13 02:12:30,871 - INFO - Pod ctest-nginx-pod-99249441 verification passed
2025-03-13 02:12:30,941 - DEBUG - [Pod ctest-nginx-pod-99249441] Cmd: echo ctest-nginx-pod-99249441 > /usr/share/nginx/html/index.html, Output:
2025-03-13 02:12:30,947 - DEBUG - Pod ctest-nginx-pod-72312214 not in running state.Currently in Pending
2025-03-13 02:12:35,954 - INFO - Pod ctest-nginx-pod-72312214 is in running state.Got IP 10.47.255.250
2025-03-13 02:12:35,971 - DEBUG - Pod ctest-nginx-pod-72312214 has vmi 98732866-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:12:35,971 - INFO - Verified pod ctest-nginx-pod-72312214 in contrail-api
2025-03-13 02:12:35,972 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:12:35,986 - DEBUG - VMI 98732866-ffb0-11ef-af8a-fa163e56332f is active in agent 10.0.0.28
2025-03-13 02:12:35,986 - INFO - Verified Pod ctest-nginx-pod-72312214 in agent 10.0.0.28
2025-03-13 02:12:35,986 - DEBUG - Requesting: http://10.0.0.12:8108/Snh_VirtualMachineDatabaseList?x=34b36a5c-aad1-4fb0-a099-e66776188fc7
2025-03-13 02:12:35,992 - INFO - Pod ctest-nginx-pod-72312214 with uuid 34b36a5c-aad1-4fb0-a099-e66776188fc7 found in kube manager
2025-03-13 02:12:35,992 - INFO - Pod ctest-nginx-pod-72312214 verification passed
2025-03-13 02:12:36,047 - DEBUG - [Pod ctest-nginx-pod-72312214] Cmd: echo ctest-nginx-pod-72312214 > /usr/share/nginx/html/index.html, Output:
2025-03-13 02:12:36,054 - INFO - Pod ctest-busybox-pod-90941759 is in running state.Got IP 10.47.255.248
2025-03-13 02:12:36,070 - DEBUG - Pod ctest-busybox-pod-90941759 has vmi 9b9b14e0-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:12:36,070 - INFO - Verified pod ctest-busybox-pod-90941759 in contrail-api
2025-03-13 02:12:36,070 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:12:36,083 - DEBUG - VMI 9b9b14e0-ffb0-11ef-af8a-fa163e56332f is active in agent 10.0.0.28
2025-03-13 02:12:36,084 - INFO - Verified Pod ctest-busybox-pod-90941759 in agent 10.0.0.28
2025-03-13 02:12:36,085 - DEBUG - Requesting: http://10.0.0.12:8108/Snh_VirtualMachineDatabaseList?x=494bbf83-9a46-4234-bef9-30b43736456e
2025-03-13 02:12:36,091 - INFO - Pod ctest-busybox-pod-90941759 with uuid 494bbf83-9a46-4234-bef9-30b43736456e found in kube manager
2025-03-13 02:12:36,091 - INFO - Pod ctest-busybox-pod-90941759 verification passed
2025-03-13 02:12:36,160 - DEBUG - [Pod ctest-busybox-pod-90941759] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-72312214
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-03-13 02:12:36,161 - DEBUG - [Pod Pod ctest-busybox-pod-90941759] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-03-13 02:12:36,161 - DEBUG - [Pod Pod ctest-busybox-pod-90941759] Cmd output: ctest-nginx-pod-72312214
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-03-13 02:12:36,224 - DEBUG - [Pod ctest-busybox-pod-90941759] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-72312214
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-03-13 02:12:36,224 - DEBUG - [Pod Pod ctest-busybox-pod-90941759] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-03-13 02:12:36,224 - DEBUG - [Pod Pod ctest-busybox-pod-90941759] Cmd output: ctest-nginx-pod-72312214
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-03-13 02:12:36,289 - DEBUG - [Pod ctest-busybox-pod-90941759] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-99249441
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-03-13 02:12:36,289 - DEBUG - [Pod Pod ctest-busybox-pod-90941759] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-03-13 02:12:36,289 - DEBUG - [Pod Pod ctest-busybox-pod-90941759] Cmd output: ctest-nginx-pod-99249441
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-03-13 02:12:36,289 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-99249441': 1, 'ctest-nginx-pod-72312214': 2}
2025-03-13 02:12:36,289 - WARNING - Nothing to delete parallely
2025-03-13 02:12:36,289 - INFO - Deleting pod default:ctest-busybox-pod-90941759
2025-03-13 02:12:36,303 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=494bbf83-9a46-4234-bef9-30b43736456e
2025-03-13 02:12:36,311 - WARNING - Pod uuid 494bbf83-9a46-4234-bef9-30b43736456e is still seen in agent 10.0.0.28 VM list
2025-03-13 02:12:41,313 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=494bbf83-9a46-4234-bef9-30b43736456e
2025-03-13 02:12:41,321 - WARNING - Pod uuid 494bbf83-9a46-4234-bef9-30b43736456e is still seen in agent 10.0.0.28 VM list
2025-03-13 02:12:46,322 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=494bbf83-9a46-4234-bef9-30b43736456e
2025-03-13 02:12:46,331 - WARNING - Pod uuid 494bbf83-9a46-4234-bef9-30b43736456e is still seen in agent 10.0.0.28 VM list
2025-03-13 02:12:51,331 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=494bbf83-9a46-4234-bef9-30b43736456e
2025-03-13 02:12:51,339 - WARNING - Pod uuid 494bbf83-9a46-4234-bef9-30b43736456e is still seen in agent 10.0.0.28 VM list
2025-03-13 02:12:56,340 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=494bbf83-9a46-4234-bef9-30b43736456e
2025-03-13 02:12:56,349 - WARNING - Pod uuid 494bbf83-9a46-4234-bef9-30b43736456e is still seen in agent 10.0.0.28 VM list
2025-03-13 02:13:01,350 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=494bbf83-9a46-4234-bef9-30b43736456e
2025-03-13 02:13:01,361 - WARNING - Pod uuid 494bbf83-9a46-4234-bef9-30b43736456e is still seen in agent 10.0.0.28 VM list
2025-03-13 02:13:06,362 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=494bbf83-9a46-4234-bef9-30b43736456e
2025-03-13 02:13:06,375 - WARNING - Pod uuid 494bbf83-9a46-4234-bef9-30b43736456e is still seen in agent 10.0.0.28 VM list
2025-03-13 02:13:11,376 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=494bbf83-9a46-4234-bef9-30b43736456e
2025-03-13 02:13:11,384 - DEBUG - Pod 494bbf83-9a46-4234-bef9-30b43736456e is not in agent 10.0.0.28 VM list
2025-03-13 02:13:11,384 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9b9b14e0-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:13:11,394 - DEBUG - VMI 9b9b14e0-ffb0-11ef-af8a-fa163e56332f is removed from agent 10.0.0.28
2025-03-13 02:13:11,394 - INFO - Verified that pod ctest-busybox-pod-90941759 is removed in agent
2025-03-13 02:13:11,395 - INFO - Deleting Ingress : ctest-nginx-ingress-32063800
2025-03-13 02:13:11,402 - DEBUG - Requesting: http://10.0.0.12:8108/Snh_LoadbalancerDatabaseList?x=c9ef9119-b15f-4c40-b51b-66b1cbfc67a6
2025-03-13 02:13:11,408 - ERROR - Ingress ctest-nginx-ingress-32063800 with uuid c9ef9119-b15f-4c40-b51b-66b1cbfc67a6 still found in kube manager
2025-03-13 02:13:12,409 - DEBUG - Requesting: http://10.0.0.12:8108/Snh_LoadbalancerDatabaseList?x=c9ef9119-b15f-4c40-b51b-66b1cbfc67a6
2025-03-13 02:13:12,414 - INFO - Ingress ctest-nginx-ingress-32063800 with uuid c9ef9119-b15f-4c40-b51b-66b1cbfc67a6 deleted successfully from kube manager
2025-03-13 02:13:12,416 - INFO - Verifications on Ingress %s cleanup passed
2025-03-13 02:13:12,416 - INFO - Deleting the FIP pool __fip_pool_public__
2025-03-13 02:13:12,552 - DEBUG - Requesting: http://10.0.0.12:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-03-13 02:13:12,559 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-03-13 02:13:12,559 - INFO - Deleting VN __public__
2025-03-13 02:13:12,843 - DEBUG - Requesting: http://10.0.0.12:8082/routing-instance/c279f83c-38a8-4daf-9cb8-cf5eebde000d
2025-03-13 02:13:12,857 - DEBUG - Response Code: 404
2025-03-13 02:13:12,857 - DEBUG - Requesting: http://10.0.0.12:8082/domains
2025-03-13 02:13:12,866 - DEBUG - Requesting: http://10.0.0.12:8082/domain/54cacbf0-0712-46f8-9c8a-7546eb095f7b
2025-03-13 02:13:12,883 - DEBUG - Requesting: http://10.0.0.12:8082/project/b4ef722d-dac4-45a8-a477-60403d029a2d
2025-03-13 02:13:12,942 - INFO - Validated that VN __public__ is not found in API Server
2025-03-13 02:13:12,942 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-03-13 02:13:12,952 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-03-13 02:13:12,975 - DEBUG - VN __public__ is not present in Agent 10.0.0.28
2025-03-13 02:13:12,975 - INFO - Validated that VN __public__ is not in any agent
2025-03-13 02:13:12,976 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-03-13 02:13:12,990 - DEBUG - Requesting: http://10.0.0.12:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-03-13 02:13:12,997 - DEBUG - Requesting: http://10.0.0.12:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-03-13 02:13:13,003 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-03-13 02:13:13,004 - INFO - Deleting pod default:ctest-nginx-pod-72312214
2025-03-13 02:13:13,017 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=34b36a5c-aad1-4fb0-a099-e66776188fc7
2025-03-13 02:13:13,034 - WARNING - Pod uuid 34b36a5c-aad1-4fb0-a099-e66776188fc7 is still seen in agent 10.0.0.28 VM list
2025-03-13 02:13:18,042 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=34b36a5c-aad1-4fb0-a099-e66776188fc7
2025-03-13 02:13:18,052 - DEBUG - Pod 34b36a5c-aad1-4fb0-a099-e66776188fc7 is not in agent 10.0.0.28 VM list
2025-03-13 02:13:18,052 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:98732866-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:13:18,061 - DEBUG - VMI 98732866-ffb0-11ef-af8a-fa163e56332f is removed from agent 10.0.0.28
2025-03-13 02:13:18,061 - INFO - Verified that pod ctest-nginx-pod-72312214 is removed in agent
2025-03-13 02:13:18,061 - INFO - Deleting pod default:ctest-nginx-pod-99249441
2025-03-13 02:13:18,097 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=478dfad1-c620-4736-b979-dfb4cde254db
2025-03-13 02:13:18,110 - WARNING - Pod uuid 478dfad1-c620-4736-b979-dfb4cde254db is still seen in agent 10.0.0.28 VM list
2025-03-13 02:13:23,111 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=478dfad1-c620-4736-b979-dfb4cde254db
2025-03-13 02:13:23,125 - DEBUG - Pod 478dfad1-c620-4736-b979-dfb4cde254db is not in agent 10.0.0.28 VM list
2025-03-13 02:13:23,125 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:97fa5e40-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:13:23,132 - DEBUG - VMI 97fa5e40-ffb0-11ef-af8a-fa163e56332f is removed from agent 10.0.0.28
2025-03-13 02:13:23,132 - INFO - Verified that pod ctest-nginx-pod-99249441 is removed in agent
2025-03-13 02:13:23,133 - INFO - Deleting service : ctest-nginx-svc-13093608
2025-03-13 02:13:23,403 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_AgentXmppConnectionStatusReq?
2025-03-13 02:13:23,411 - DEBUG - No XMPP flaps were noticed during the test
2025-03-13 02:13:23,411 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:06]
2025-03-13 02:13:23,412 - INFO - --------------------------------------------------------------------------------
2025-03-13 02:13:29,821 - DEBUG - [10.0.0.12]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-03-13 02:13:29,975 - DEBUG - Output : cluster_project={}
2025-03-13 02:13:30,089 - DEBUG - [10.0.0.12]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-03-13 02:13:30,288 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-03-13 02:13:30,293 - DEBUG - [10.0.0.12]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-03-13 02:13:30,435 - DEBUG - Output : cluster_project={}
2025-03-13 02:13:45,554 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:13:45,641 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:13:45,693 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:13:45,790 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:13:50,861 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:13:50,942 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:13:51,060 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:13:51,146 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-03-13 02:13:51,423 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_AgentXmppConnectionStatusReq?
2025-03-13 02:13:54,216 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_ApplicationPolicySetReq?
2025-03-13 02:13:54,224 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_AclReq?
2025-03-13 02:13:54,257 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_AclReq?
2025-03-13 02:13:56,290 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_AclReq?
2025-03-13 02:16:57,039 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_AgentXmppConnectionStatusReq?
2025-03-13 02:16:57,095 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=e161cd9d-3781-4a63-99bb-b49ef4cf00b8
2025-03-13 02:16:57,114 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=9041ae7f-26cf-40c6-ba58-15273245f98e
2025-03-13 02:16:57,123 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=5926fd40-968b-4211-be70-d40ba8179884
2025-03-13 02:16:57,130 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=a883b562-9d48-4f17-8771-6aa4c550f5e1
2025-03-13 02:16:57,137 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=7a307c5e-997a-4b0d-aa76-28b98f3a7cab
2025-03-13 02:16:57,142 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=505c254f-186f-4243-aded-708c2a33b08f
2025-03-13 02:16:57,146 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=1777a0e4-87b6-49c5-868a-09305e475a2d
2025-03-13 02:16:57,152 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=9a671614-a665-4968-b7b3-6848230d2c7d
2025-03-13 02:17:02,187 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=1777a0e4-87b6-49c5-868a-09305e475a2d
2025-03-13 02:17:02,190 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=505c254f-186f-4243-aded-708c2a33b08f
2025-03-13 02:17:02,191 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=7a307c5e-997a-4b0d-aa76-28b98f3a7cab
2025-03-13 02:17:02,192 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=a883b562-9d48-4f17-8771-6aa4c550f5e1
2025-03-13 02:17:02,194 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=9a671614-a665-4968-b7b3-6848230d2c7d
2025-03-13 02:17:02,196 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=9041ae7f-26cf-40c6-ba58-15273245f98e
2025-03-13 02:17:02,199 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=e161cd9d-3781-4a63-99bb-b49ef4cf00b8
2025-03-13 02:17:02,200 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=5926fd40-968b-4211-be70-d40ba8179884
2025-03-13 02:17:02,225 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c297e46a-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:17:02,229 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c47a6bfe-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:17:02,231 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c3e917ee-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:17:02,232 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c3238218-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:17:07,229 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=5926fd40-968b-4211-be70-d40ba8179884
2025-03-13 02:17:07,230 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=505c254f-186f-4243-aded-708c2a33b08f
2025-03-13 02:17:07,232 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=a883b562-9d48-4f17-8771-6aa4c550f5e1
2025-03-13 02:17:07,238 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=9041ae7f-26cf-40c6-ba58-15273245f98e
2025-03-13 02:17:12,244 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=a883b562-9d48-4f17-8771-6aa4c550f5e1
2025-03-13 02:17:12,246 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=505c254f-186f-4243-aded-708c2a33b08f
2025-03-13 02:17:12,247 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=9041ae7f-26cf-40c6-ba58-15273245f98e
2025-03-13 02:17:12,249 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=5926fd40-968b-4211-be70-d40ba8179884
2025-03-13 02:17:17,265 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=a883b562-9d48-4f17-8771-6aa4c550f5e1
2025-03-13 02:17:17,267 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=505c254f-186f-4243-aded-708c2a33b08f
2025-03-13 02:17:17,269 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=9041ae7f-26cf-40c6-ba58-15273245f98e
2025-03-13 02:17:17,272 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=5926fd40-968b-4211-be70-d40ba8179884
2025-03-13 02:17:22,288 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=505c254f-186f-4243-aded-708c2a33b08f
2025-03-13 02:17:22,290 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=a883b562-9d48-4f17-8771-6aa4c550f5e1
2025-03-13 02:17:22,292 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=5926fd40-968b-4211-be70-d40ba8179884
2025-03-13 02:17:22,294 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=9041ae7f-26cf-40c6-ba58-15273245f98e
2025-03-13 02:17:27,310 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=505c254f-186f-4243-aded-708c2a33b08f
2025-03-13 02:17:27,312 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=a883b562-9d48-4f17-8771-6aa4c550f5e1
2025-03-13 02:17:27,314 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=9041ae7f-26cf-40c6-ba58-15273245f98e
2025-03-13 02:17:27,315 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=5926fd40-968b-4211-be70-d40ba8179884
2025-03-13 02:17:32,394 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=505c254f-186f-4243-aded-708c2a33b08f
2025-03-13 02:17:32,396 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=9041ae7f-26cf-40c6-ba58-15273245f98e
2025-03-13 02:17:32,398 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=a883b562-9d48-4f17-8771-6aa4c550f5e1
2025-03-13 02:17:32,399 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_VmListReq?uuid=5926fd40-968b-4211-be70-d40ba8179884
2025-03-13 02:17:32,415 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c61b98b6-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:17:32,417 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c5138e74-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:17:32,418 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c59a9e6e-ffb0-11ef-af8a-fa163e56332f
2025-03-13 02:17:32,422 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c6a5386e-ffb0-11ef-af8a-fa163e56332f