2025-05-13 08:17:55,365 - DEBUG - [10.0.0.249]: Running cmd : hostname
2025-05-13 08:17:55,610 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1225-1
2025-05-13 08:17:55,610 - DEBUG - [10.0.0.249]: Running cmd : hostname -f
2025-05-13 08:17:55,644 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1225-1
2025-05-13 08:17:55,644 - DEBUG - [10.0.0.249]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-05-13 08:17:55,698 - DEBUG - Output : NAMES
contrail_test_ZmKhptiP7
contrail-kubernetes-master-kubemanager-1
analytics_snmp-snmp-collector-1
analytics_snmp-topology-1
analytics_snmp-nodemgr-1
analytics_alarm-nodemgr-1
analytics_alarm-alarm-gen-1
analytics_alarm-kafka-1
analytics-api-1
analytics-collector-1
analytics-nodemgr-1
analytics_database-nodemgr-1
analytics_database-query-engine-1
analytics_database-cassandra-1
redis-redis-1
webui-web-1
webui-job-1
control-named-1
control-nodemgr-1
control-dns-1
control-control-1
config_database-zookeeper-1
config_database-cassandra-1
config_database-rabbitmq-1
config_database-nodemgr-1
config_api-devicemgr-1
config_api-nodemgr-1
config_api-api-1
config_api-schema-1
config_api-svcmonitor-1
config_api-dnsmasq-1
2025-05-13 08:17:55,698 - DEBUG - [10.0.0.249]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-05-13 08:17:55,772 - DEBUG - Output : 10.0.0.249/24
172.17.0.1/16
2025-05-13 08:17:55,772 - DEBUG - [10.0.0.249]: Running cmd : getent hosts 10.0.0.249 | head -n 1 | awk '{print $2}'
2025-05-13 08:17:55,817 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1225-1
2025-05-13 08:17:55,917 - DEBUG - [10.0.0.19]: Running cmd : hostname
2025-05-13 08:17:56,130 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1225-1
2025-05-13 08:17:56,131 - DEBUG - [10.0.0.19]: Running cmd : hostname -f
2025-05-13 08:17:56,154 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1225-1
2025-05-13 08:17:56,154 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-05-13 08:17:56,198 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2025-05-13 08:17:56,198 - DEBUG - [10.0.0.19]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-05-13 08:17:56,223 - DEBUG - Output : 10.0.0.19/24
2025-05-13 08:17:56,223 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.0.0.19 | head -n 1 | awk '{print $2}'
2025-05-13 08:17:56,298 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1225-1
2025-05-13 08:17:56,366 - INFO - Using existing project ['default-domain', 'k8s-default'](e9d4682f-c55b-47eb-8534-36b8c790c646)
2025-05-13 08:17:56,466 - INFO - Using existing project ['default-domain', 'k8s-default'](e9d4682f-c55b-47eb-8534-36b8c790c646)
2025-05-13 08:17:56,600 - INFO - ================================================================================
2025-05-13 08:17:56,600 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-05-13 08:17:56,601 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-05-13 08:17:56,836 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-05-13 08:17:56,844 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.0.0.249': '0'}} with
2025-05-13 08:17:56,844 - INFO - Initial checks done. Running the testcase now
2025-05-13 08:17:56,845 - INFO -
2025-05-13 08:17:56,849 - INFO - Namespace default already exists
2025-05-13 08:17:56,849 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-05-13 08:17:56,948 - DEBUG - Output : cluster_project={}
2025-05-13 08:17:56,990 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-05-13 08:17:56,990 - DEBUG - Project uuid in Contrail is e9d4682f-c55b-47eb-8534-36b8c790c646
2025-05-13 08:17:56,990 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-05-13 08:17:57,106 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-05-13 08:17:57,106 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_NamespaceDatabaseList?namespace_uuid=920a7236-f35d-4d6a-b81c-783385debda8
2025-05-13 08:17:57,111 - INFO - Namespace default with uuid 920a7236-f35d-4d6a-b81c-783385debda8 found in kube manager
2025-05-13 08:17:57,111 - INFO - Namespace default verification passed
2025-05-13 08:17:57,113 - DEBUG - Service ctest-nginx-svc-72428758 not present
2025-05-13 08:17:57,114 - INFO - Creating service ctest-nginx-svc-72428758
2025-05-13 08:17:57,130 - INFO - Created Service ctest-nginx-svc-72428758
2025-05-13 08:17:57,133 - DEBUG - Pod ctest-nginx-pod-62091953 not present
2025-05-13 08:17:57,133 - INFO - Creating Pod ctest-nginx-pod-62091953
2025-05-13 08:17:57,181 - DEBUG - Node for Pod ctest-nginx-pod-62091953 not yet populated
2025-05-13 08:18:00,186 - DEBUG - Pod : ctest-nginx-pod-62091953 UUID is b41b38b7-0dbc-40f6-88da-5e817f1987b0
2025-05-13 08:18:00,188 - DEBUG - Pod ctest-nginx-pod-84266477 not present
2025-05-13 08:18:00,189 - INFO - Creating Pod ctest-nginx-pod-84266477
2025-05-13 08:18:00,229 - DEBUG - Pod : ctest-nginx-pod-84266477 UUID is 8ea2713b-5fba-45cc-bab7-3de0fada5d31
2025-05-13 08:18:00,414 - INFO - Created VN __public__, UUID :15b37e7e-9bc0-4ac2-8404-55cadb3847e9
2025-05-13 08:18:00,579 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2025-05-13 08:18:00,586 - DEBUG - Requesting: http://10.0.0.249:8082/domain/a7fbc4e8-0459-4810-9ac6-799410438f3f
2025-05-13 08:18:00,604 - DEBUG - Requesting: http://10.0.0.249:8082/project/e9d4682f-c55b-47eb-8534-36b8c790c646
2025-05-13 08:18:00,663 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/15b37e7e-9bc0-4ac2-8404-55cadb3847e9
2025-05-13 08:18:00,677 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/15b37e7e-9bc0-4ac2-8404-55cadb3847e9
2025-05-13 08:18:00,689 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/a1cc88a5-e871-46e3-9280-2c0f6242cd7e
2025-05-13 08:18:00,696 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/a1cc88a5-e871-46e3-9280-2c0f6242cd7e
2025-05-13 08:18:00,722 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/cb4e2ba8-ad80-4bad-af0a-6418e35e48ad
2025-05-13 08:18:00,731 - DEBUG - Route Targets: ['target:64512:8000007']
2025-05-13 08:18:00,731 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/15b37e7e-9bc0-4ac2-8404-55cadb3847e9
2025-05-13 08:18:00,753 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/a1cc88a5-e871-46e3-9280-2c0f6242cd7e
2025-05-13 08:18:00,773 - INFO - Verified VN network id 14 for VN 15b37e7e-9bc0-4ac2-8404-55cadb3847e9
2025-05-13 08:18:00,773 - INFO - Verifications in API Server for VN __public__ passed
2025-05-13 08:18:00,774 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/15b37e7e-9bc0-4ac2-8404-55cadb3847e9
2025-05-13 08:18:00,782 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/a1cc88a5-e871-46e3-9280-2c0f6242cd7e
2025-05-13 08:18:00,789 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/a1cc88a5-e871-46e3-9280-2c0f6242cd7e
2025-05-13 08:18:00,795 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/cb4e2ba8-ad80-4bad-af0a-6418e35e48ad
2025-05-13 08:18:00,802 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-05-13 08:18:00,809 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1563732577871743682', 'uuid-lslong': '9512822642662983657'}, 'enable': 'true', 'created': '2025-05-13T08:18:00', 'last-modified': '2025-05-13T08:18:00', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.344840'}
2025-05-13 08:18:00,809 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-05-13 08:18:00,818 - DEBUG - Route Targets: ['target:64512:8000007']
2025-05-13 08:18:00,818 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-05-13 08:18:00,819 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-05-13 08:18:00,819 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2025-05-13 08:18:00,828 - DEBUG - Requesting: http://10.0.0.249:8082/domain/a7fbc4e8-0459-4810-9ac6-799410438f3f
2025-05-13 08:18:00,859 - DEBUG - Requesting: http://10.0.0.249:8082/project/e9d4682f-c55b-47eb-8534-36b8c790c646
2025-05-13 08:18:00,900 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/15b37e7e-9bc0-4ac2-8404-55cadb3847e9
2025-05-13 08:18:00,909 - DEBUG - =>VN __public__ has no policy to be verified
2025-05-13 08:18:00,909 - DEBUG - Verifying the vn in opserver
2025-05-13 08:18:00,909 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.249
2025-05-13 08:18:00,909 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks
2025-05-13 08:18:00,915 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-05-13 08:18:00,915 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-05-13 08:18:00,915 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-05-13 08:18:00,923 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-05-13 08:18:00,931 - DEBUG - Do not have enough data to verify VN in agent
2025-05-13 08:18:00,931 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-05-13 08:18:00,937 - DEBUG - VRF ids for VN __public__: {}
2025-05-13 08:18:01,051 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-05-13 08:18:01,151 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2025-05-13 08:18:01,156 - DEBUG - Requesting: http://10.0.0.249:8082/domain/a7fbc4e8-0459-4810-9ac6-799410438f3f
2025-05-13 08:18:01,164 - DEBUG - Requesting: http://10.0.0.249:8082/project/e9d4682f-c55b-47eb-8534-36b8c790c646
2025-05-13 08:18:01,227 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/15b37e7e-9bc0-4ac2-8404-55cadb3847e9
2025-05-13 08:18:01,238 - DEBUG - Requesting: http://10.0.0.249:8082/floating-ip-pool/b2dfe6f4-a638-4864-994f-c51b155de4de
2025-05-13 08:18:01,244 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2025-05-13 08:18:01,249 - DEBUG - Requesting: http://10.0.0.249:8082/domain/a7fbc4e8-0459-4810-9ac6-799410438f3f
2025-05-13 08:18:01,258 - DEBUG - Requesting: http://10.0.0.249:8082/project/e9d4682f-c55b-47eb-8534-36b8c790c646
2025-05-13 08:18:01,299 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/15b37e7e-9bc0-4ac2-8404-55cadb3847e9
2025-05-13 08:18:01,309 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-05-13 08:18:01,309 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-05-13 08:18:01,320 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-05-13 08:18:01,329 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-05-13 08:18:01,329 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-05-13 08:18:01,333 - DEBUG - Ingress ctest-nginx-ingress-04034804 not present
2025-05-13 08:18:01,334 - INFO - Creating Ingress ctest-nginx-ingress-04034804
2025-05-13 08:18:01,340 - INFO - Created Ingress ctest-nginx-ingress-04034804
2025-05-13 08:18:01,343 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-04034804
2025-05-13 08:18:04,347 - DEBUG - For Ingress ctest-nginx-ingress-04034804, Cluster IP: 10.47.255.249, External IPs ['164.157.23.252']
2025-05-13 08:18:04,347 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-04034804
2025-05-13 08:18:04,347 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=10e14de5-45ea-4a5c-930b-c827efaee347
2025-05-13 08:18:04,352 - INFO - Ingress ctest-nginx-ingress-04034804 with uuid 10e14de5-45ea-4a5c-930b-c827efaee347 found in kube manager
2025-05-13 08:18:04,352 - INFO - Ingress ctest-nginx-ingress-04034804 verification passed
2025-05-13 08:18:04,354 - DEBUG - Pod ctest-busybox-pod-06273662 not present
2025-05-13 08:18:04,354 - INFO - Creating Pod ctest-busybox-pod-06273662
2025-05-13 08:18:04,366 - DEBUG - Node for Pod ctest-busybox-pod-06273662 not yet populated
2025-05-13 08:18:07,372 - DEBUG - Pod : ctest-busybox-pod-06273662 UUID is 5b0800f4-a014-490e-82e2-7b864c674700
2025-05-13 08:18:07,376 - DEBUG - Pod ctest-nginx-pod-62091953 not in running state.Currently in Pending
2025-05-13 08:18:12,381 - DEBUG - Pod ctest-nginx-pod-62091953 not in running state.Currently in Pending
2025-05-13 08:18:17,388 - INFO - Pod ctest-nginx-pod-62091953 is in running state.Got IP 10.47.255.251
2025-05-13 08:18:17,443 - DEBUG - Pod ctest-nginx-pod-62091953 has vmi c78164a4-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:18:17,443 - INFO - Verified pod ctest-nginx-pod-62091953 in contrail-api
2025-05-13 08:18:17,443 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:18:17,457 - DEBUG - VMI c78164a4-2fd2-11f0-a637-fa163e2b49a7 is active in agent 10.0.0.19
2025-05-13 08:18:17,457 - INFO - Verified Pod ctest-nginx-pod-62091953 in agent 10.0.0.19
2025-05-13 08:18:17,457 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=b41b38b7-0dbc-40f6-88da-5e817f1987b0
2025-05-13 08:18:17,461 - INFO - Pod ctest-nginx-pod-62091953 with uuid b41b38b7-0dbc-40f6-88da-5e817f1987b0 found in kube manager
2025-05-13 08:18:17,461 - INFO - Pod ctest-nginx-pod-62091953 verification passed
2025-05-13 08:18:17,550 - DEBUG - [Pod ctest-nginx-pod-62091953] Cmd: echo ctest-nginx-pod-62091953 > /usr/share/nginx/html/index.html, Output:
2025-05-13 08:18:17,556 - DEBUG - Pod ctest-nginx-pod-84266477 not in running state.Currently in Pending
2025-05-13 08:18:22,562 - INFO - Pod ctest-nginx-pod-84266477 is in running state.Got IP 10.47.255.250
2025-05-13 08:18:22,576 - DEBUG - Pod ctest-nginx-pod-84266477 has vmi c8a071c2-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:18:22,576 - INFO - Verified pod ctest-nginx-pod-84266477 in contrail-api
2025-05-13 08:18:22,576 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:18:22,588 - DEBUG - VMI c8a071c2-2fd2-11f0-a637-fa163e2b49a7 is active in agent 10.0.0.19
2025-05-13 08:18:22,588 - INFO - Verified Pod ctest-nginx-pod-84266477 in agent 10.0.0.19
2025-05-13 08:18:22,588 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=8ea2713b-5fba-45cc-bab7-3de0fada5d31
2025-05-13 08:18:22,593 - INFO - Pod ctest-nginx-pod-84266477 with uuid 8ea2713b-5fba-45cc-bab7-3de0fada5d31 found in kube manager
2025-05-13 08:18:22,593 - INFO - Pod ctest-nginx-pod-84266477 verification passed
2025-05-13 08:18:22,643 - DEBUG - [Pod ctest-nginx-pod-84266477] Cmd: echo ctest-nginx-pod-84266477 > /usr/share/nginx/html/index.html, Output:
2025-05-13 08:18:22,648 - INFO - Pod ctest-busybox-pod-06273662 is in running state.Got IP 10.47.255.248
2025-05-13 08:18:22,662 - DEBUG - Pod ctest-busybox-pod-06273662 has vmi cb18ffb4-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:18:22,662 - INFO - Verified pod ctest-busybox-pod-06273662 in contrail-api
2025-05-13 08:18:22,663 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:18:22,673 - DEBUG - VMI cb18ffb4-2fd2-11f0-a637-fa163e2b49a7 is active in agent 10.0.0.19
2025-05-13 08:18:22,674 - INFO - Verified Pod ctest-busybox-pod-06273662 in agent 10.0.0.19
2025-05-13 08:18:22,674 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=5b0800f4-a014-490e-82e2-7b864c674700
2025-05-13 08:18:22,679 - INFO - Pod ctest-busybox-pod-06273662 with uuid 5b0800f4-a014-490e-82e2-7b864c674700 found in kube manager
2025-05-13 08:18:22,679 - INFO - Pod ctest-busybox-pod-06273662 verification passed
2025-05-13 08:18:22,765 - DEBUG - [Pod ctest-busybox-pod-06273662] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-62091953
2025-05-13 08:18:22,766 - DEBUG - [Pod Pod ctest-busybox-pod-06273662] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-05-13 08:18:22,766 - DEBUG - [Pod Pod ctest-busybox-pod-06273662] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-62091953
2025-05-13 08:18:22,831 - DEBUG - [Pod ctest-busybox-pod-06273662] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-84266477
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-05-13 08:18:22,831 - DEBUG - [Pod Pod ctest-busybox-pod-06273662] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-05-13 08:18:22,831 - DEBUG - [Pod Pod ctest-busybox-pod-06273662] Cmd output: ctest-nginx-pod-84266477
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-05-13 08:18:22,831 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-62091953': 1, 'ctest-nginx-pod-84266477': 1}
2025-05-13 08:18:22,831 - WARNING - Nothing to delete parallely
2025-05-13 08:18:22,831 - INFO - Deleting pod default:ctest-busybox-pod-06273662
2025-05-13 08:18:22,842 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=5b0800f4-a014-490e-82e2-7b864c674700
2025-05-13 08:18:22,852 - WARNING - Pod uuid 5b0800f4-a014-490e-82e2-7b864c674700 is still seen in agent 10.0.0.19 VM list
2025-05-13 08:18:27,853 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=5b0800f4-a014-490e-82e2-7b864c674700
2025-05-13 08:18:27,861 - WARNING - Pod uuid 5b0800f4-a014-490e-82e2-7b864c674700 is still seen in agent 10.0.0.19 VM list
2025-05-13 08:18:32,862 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=5b0800f4-a014-490e-82e2-7b864c674700
2025-05-13 08:18:32,870 - WARNING - Pod uuid 5b0800f4-a014-490e-82e2-7b864c674700 is still seen in agent 10.0.0.19 VM list
2025-05-13 08:18:37,871 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=5b0800f4-a014-490e-82e2-7b864c674700
2025-05-13 08:18:37,878 - WARNING - Pod uuid 5b0800f4-a014-490e-82e2-7b864c674700 is still seen in agent 10.0.0.19 VM list
2025-05-13 08:18:42,880 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=5b0800f4-a014-490e-82e2-7b864c674700
2025-05-13 08:18:42,887 - WARNING - Pod uuid 5b0800f4-a014-490e-82e2-7b864c674700 is still seen in agent 10.0.0.19 VM list
2025-05-13 08:18:47,887 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=5b0800f4-a014-490e-82e2-7b864c674700
2025-05-13 08:18:47,894 - WARNING - Pod uuid 5b0800f4-a014-490e-82e2-7b864c674700 is still seen in agent 10.0.0.19 VM list
2025-05-13 08:18:52,895 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=5b0800f4-a014-490e-82e2-7b864c674700
2025-05-13 08:18:52,904 - WARNING - Pod uuid 5b0800f4-a014-490e-82e2-7b864c674700 is still seen in agent 10.0.0.19 VM list
2025-05-13 08:18:57,905 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=5b0800f4-a014-490e-82e2-7b864c674700
2025-05-13 08:18:57,912 - DEBUG - Pod 5b0800f4-a014-490e-82e2-7b864c674700 is not in agent 10.0.0.19 VM list
2025-05-13 08:18:57,912 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:cb18ffb4-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:18:57,918 - DEBUG - VMI cb18ffb4-2fd2-11f0-a637-fa163e2b49a7 is removed from agent 10.0.0.19
2025-05-13 08:18:57,918 - INFO - Verified that pod ctest-busybox-pod-06273662 is removed in agent
2025-05-13 08:18:57,918 - INFO - Deleting Ingress : ctest-nginx-ingress-04034804
2025-05-13 08:18:57,926 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=10e14de5-45ea-4a5c-930b-c827efaee347
2025-05-13 08:18:57,932 - ERROR - Ingress ctest-nginx-ingress-04034804 with uuid 10e14de5-45ea-4a5c-930b-c827efaee347 still found in kube manager
2025-05-13 08:18:58,933 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=10e14de5-45ea-4a5c-930b-c827efaee347
2025-05-13 08:18:58,937 - INFO - Ingress ctest-nginx-ingress-04034804 with uuid 10e14de5-45ea-4a5c-930b-c827efaee347 deleted successfully from kube manager
2025-05-13 08:18:58,938 - INFO - Verifications on Ingress %s cleanup passed
2025-05-13 08:18:58,938 - INFO - Deleting the FIP pool __fip_pool_public__
2025-05-13 08:18:59,052 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-05-13 08:18:59,058 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-05-13 08:18:59,058 - INFO - Deleting VN __public__
2025-05-13 08:18:59,235 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/a1cc88a5-e871-46e3-9280-2c0f6242cd7e
2025-05-13 08:18:59,243 - DEBUG - Response Code: 404
2025-05-13 08:18:59,244 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2025-05-13 08:18:59,253 - DEBUG - Requesting: http://10.0.0.249:8082/domain/a7fbc4e8-0459-4810-9ac6-799410438f3f
2025-05-13 08:18:59,262 - DEBUG - Requesting: http://10.0.0.249:8082/project/e9d4682f-c55b-47eb-8534-36b8c790c646
2025-05-13 08:18:59,335 - INFO - Validated that VN __public__ is not found in API Server
2025-05-13 08:18:59,336 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-05-13 08:18:59,344 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-05-13 08:18:59,352 - DEBUG - VN __public__ is not present in Agent 10.0.0.19
2025-05-13 08:18:59,352 - INFO - Validated that VN __public__ is not in any agent
2025-05-13 08:18:59,353 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-05-13 08:18:59,360 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-05-13 08:18:59,369 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-05-13 08:18:59,375 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-05-13 08:18:59,376 - INFO - Deleting pod default:ctest-nginx-pod-84266477
2025-05-13 08:18:59,388 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=8ea2713b-5fba-45cc-bab7-3de0fada5d31
2025-05-13 08:18:59,414 - WARNING - Pod uuid 8ea2713b-5fba-45cc-bab7-3de0fada5d31 is still seen in agent 10.0.0.19 VM list
2025-05-13 08:19:04,415 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=8ea2713b-5fba-45cc-bab7-3de0fada5d31
2025-05-13 08:19:04,422 - DEBUG - Pod 8ea2713b-5fba-45cc-bab7-3de0fada5d31 is not in agent 10.0.0.19 VM list
2025-05-13 08:19:04,422 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c8a071c2-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:19:04,428 - DEBUG - VMI c8a071c2-2fd2-11f0-a637-fa163e2b49a7 is removed from agent 10.0.0.19
2025-05-13 08:19:04,428 - INFO - Verified that pod ctest-nginx-pod-84266477 is removed in agent
2025-05-13 08:19:04,428 - INFO - Deleting pod default:ctest-nginx-pod-62091953
2025-05-13 08:19:04,438 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=b41b38b7-0dbc-40f6-88da-5e817f1987b0
2025-05-13 08:19:04,454 - WARNING - Pod uuid b41b38b7-0dbc-40f6-88da-5e817f1987b0 is still seen in agent 10.0.0.19 VM list
2025-05-13 08:19:09,455 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=b41b38b7-0dbc-40f6-88da-5e817f1987b0
2025-05-13 08:19:09,463 - DEBUG - Pod b41b38b7-0dbc-40f6-88da-5e817f1987b0 is not in agent 10.0.0.19 VM list
2025-05-13 08:19:09,463 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c78164a4-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:19:09,470 - DEBUG - VMI c78164a4-2fd2-11f0-a637-fa163e2b49a7 is removed from agent 10.0.0.19
2025-05-13 08:19:09,470 - INFO - Verified that pod ctest-nginx-pod-62091953 is removed in agent
2025-05-13 08:19:09,470 - INFO - Deleting service : ctest-nginx-svc-72428758
2025-05-13 08:19:09,660 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-05-13 08:19:09,666 - DEBUG - No XMPP flaps were noticed during the test
2025-05-13 08:19:09,666 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:13]
2025-05-13 08:19:09,666 - INFO - --------------------------------------------------------------------------------
2025-05-13 08:19:16,011 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-05-13 08:19:16,121 - DEBUG - Output : cluster_project={}
2025-05-13 08:19:16,211 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-05-13 08:19:16,358 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-05-13 08:19:16,362 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-05-13 08:19:16,552 - DEBUG - Output : cluster_project={}
2025-05-13 08:19:26,623 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:19:31,703 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:19:31,743 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:19:31,824 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:19:36,886 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:19:36,984 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:19:37,166 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:19:37,218 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-13 08:19:37,542 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-05-13 08:19:40,157 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_ApplicationPolicySetReq?
2025-05-13 08:19:40,164 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AclReq?
2025-05-13 08:19:40,197 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AclReq?
2025-05-13 08:19:42,233 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AclReq?
2025-05-13 08:22:50,284 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-05-13 08:22:50,316 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=c72751e2-3777-47e5-8fee-e25b19974a0e
2025-05-13 08:22:50,343 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=fc33ded6-53b4-4570-99e7-86ecea561385
2025-05-13 08:22:50,351 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=984a3d8c-601a-423b-bf93-755c94d37737
2025-05-13 08:22:50,356 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=62ea060c-595f-4f53-ae9e-9d263a6fe6f6
2025-05-13 08:22:50,360 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=67cbd6bb-719b-43b5-8c12-42dd1f92802c
2025-05-13 08:22:50,369 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1e791845-bc2a-40d9-b31e-67dae45126f5
2025-05-13 08:22:50,372 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1d3ed379-61b1-4361-9313-afa3f4366672
2025-05-13 08:22:50,379 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=0c4fb16b-7206-4fbb-ad27-77069fd07359
2025-05-13 08:22:55,328 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=c72751e2-3777-47e5-8fee-e25b19974a0e
2025-05-13 08:22:55,335 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f6905822-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:22:55,384 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=fc33ded6-53b4-4570-99e7-86ecea561385
2025-05-13 08:22:55,443 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1e791845-bc2a-40d9-b31e-67dae45126f5
2025-05-13 08:22:55,445 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1d3ed379-61b1-4361-9313-afa3f4366672
2025-05-13 08:22:55,446 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=984a3d8c-601a-423b-bf93-755c94d37737
2025-05-13 08:22:55,447 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=67cbd6bb-719b-43b5-8c12-42dd1f92802c
2025-05-13 08:22:55,449 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=0c4fb16b-7206-4fbb-ad27-77069fd07359
2025-05-13 08:22:55,450 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=62ea060c-595f-4f53-ae9e-9d263a6fe6f6
2025-05-13 08:22:55,468 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f81975b6-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:22:55,471 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f7a134fc-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:22:55,474 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f7053228-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:23:00,391 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=fc33ded6-53b4-4570-99e7-86ecea561385
2025-05-13 08:23:00,474 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=62ea060c-595f-4f53-ae9e-9d263a6fe6f6
2025-05-13 08:23:00,476 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1d3ed379-61b1-4361-9313-afa3f4366672
2025-05-13 08:23:00,477 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=67cbd6bb-719b-43b5-8c12-42dd1f92802c
2025-05-13 08:23:05,399 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=fc33ded6-53b4-4570-99e7-86ecea561385
2025-05-13 08:23:05,489 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=62ea060c-595f-4f53-ae9e-9d263a6fe6f6
2025-05-13 08:23:05,490 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1d3ed379-61b1-4361-9313-afa3f4366672
2025-05-13 08:23:05,492 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=67cbd6bb-719b-43b5-8c12-42dd1f92802c
2025-05-13 08:23:10,406 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=fc33ded6-53b4-4570-99e7-86ecea561385
2025-05-13 08:23:10,504 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=62ea060c-595f-4f53-ae9e-9d263a6fe6f6
2025-05-13 08:23:10,506 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=67cbd6bb-719b-43b5-8c12-42dd1f92802c
2025-05-13 08:23:10,507 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1d3ed379-61b1-4361-9313-afa3f4366672
2025-05-13 08:23:15,414 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=fc33ded6-53b4-4570-99e7-86ecea561385
2025-05-13 08:23:15,519 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=62ea060c-595f-4f53-ae9e-9d263a6fe6f6
2025-05-13 08:23:15,521 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=67cbd6bb-719b-43b5-8c12-42dd1f92802c
2025-05-13 08:23:15,522 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1d3ed379-61b1-4361-9313-afa3f4366672
2025-05-13 08:23:20,422 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=fc33ded6-53b4-4570-99e7-86ecea561385
2025-05-13 08:23:20,538 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=62ea060c-595f-4f53-ae9e-9d263a6fe6f6
2025-05-13 08:23:20,540 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1d3ed379-61b1-4361-9313-afa3f4366672
2025-05-13 08:23:20,541 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=67cbd6bb-719b-43b5-8c12-42dd1f92802c
2025-05-13 08:23:25,430 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=fc33ded6-53b4-4570-99e7-86ecea561385
2025-05-13 08:23:25,436 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f9e59dd4-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:23:25,552 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=1d3ed379-61b1-4361-9313-afa3f4366672
2025-05-13 08:23:25,554 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=62ea060c-595f-4f53-ae9e-9d263a6fe6f6
2025-05-13 08:23:25,559 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_VmListReq?uuid=67cbd6bb-719b-43b5-8c12-42dd1f92802c
2025-05-13 08:23:25,563 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f9100a34-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:23:25,565 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f89f758a-2fd2-11f0-a637-fa163e2b49a7
2025-05-13 08:23:25,567 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f97af43e-2fd2-11f0-a637-fa163e2b49a7