2025-09-15 11:18:31,755 - DEBUG - [10.0.0.19]: Running cmd : hostname
2025-09-15 11:18:31,995 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1478-1
2025-09-15 11:18:31,996 - DEBUG - [10.0.0.19]: Running cmd : hostname -f
2025-09-15 11:18:32,029 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1478-1
2025-09-15 11:18:32,029 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-09-15 11:18:32,093 - DEBUG - Output : NAMES
contrail_test_XusvhqRqH
contrail-kubernetes-master-kubemanager-1
analytics_database-nodemgr-1
analytics_database-cassandra-1
analytics_database-query-engine-1
analytics_snmp-topology-1
analytics_snmp-snmp-collector-1
analytics_snmp-nodemgr-1
analytics_alarm-alarm-gen-1
analytics_alarm-nodemgr-1
analytics_alarm-kafka-1
analytics-api-1
analytics-collector-1
analytics-nodemgr-1
redis-redis-1
webui-web-1
webui-job-1
control-named-1
control-nodemgr-1
control-control-1
control-dns-1
config_database-rabbitmq-1
config_database-nodemgr-1
config_database-cassandra-1
config_database-zookeeper-1
config_api-devicemgr-1
config_api-nodemgr-1
config_api-svcmonitor-1
config_api-dnsmasq-1
config_api-schema-1
config_api-api-1
2025-09-15 11:18:32,093 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-09-15 11:18:32,158 - DEBUG - Output : 127.0.0.1/8
10.0.0.19/24
252.19.0.1/8
10.219.91.1/24
172.17.0.1/16
2025-09-15 11:18:32,158 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.0.0.19 | head -n 1 | awk '{print $2}'
2025-09-15 11:18:32,192 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1478-1
2025-09-15 11:18:32,290 - DEBUG - [10.0.0.137]: Running cmd : hostname
2025-09-15 11:18:32,543 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1478-1
2025-09-15 11:18:32,543 - DEBUG - [10.0.0.137]: Running cmd : hostname -f
2025-09-15 11:18:32,578 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1478-1
2025-09-15 11:18:32,578 - DEBUG - [10.0.0.137]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-09-15 11:18:32,666 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2025-09-15 11:18:32,666 - DEBUG - [10.0.0.137]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-09-15 11:18:32,701 - DEBUG - Output : 10.0.0.137/24
2025-09-15 11:18:32,701 - DEBUG - [10.0.0.137]: Running cmd : getent hosts 10.0.0.137 | head -n 1 | awk '{print $2}'
2025-09-15 11:18:32,735 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1478-1
2025-09-15 11:18:32,799 - INFO - Using existing project ['default-domain', 'k8s-default'](c7452f82-ea73-460a-b135-7f161da7aa83)
2025-09-15 11:18:32,858 - INFO - Using existing project ['default-domain', 'k8s-default'](c7452f82-ea73-460a-b135-7f161da7aa83)
2025-09-15 11:18:33,005 - INFO - ================================================================================
2025-09-15 11:18:33,005 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-09-15 11:18:33,005 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-09-15 11:18:33,211 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_AgentXmppConnectionStatusReq?
2025-09-15 11:18:33,219 - DEBUG - Nothing to compare xmpp stats {'10.0.0.137': {'10.0.0.19': '0'}} with
2025-09-15 11:18:33,219 - INFO - Initial checks done. Running the testcase now
2025-09-15 11:18:33,219 - INFO -
2025-09-15 11:18:33,227 - INFO - Namespace default already exists
2025-09-15 11:18:33,227 - DEBUG - [10.0.0.19]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-09-15 11:18:33,333 - DEBUG - Output : cluster_project={}
2025-09-15 11:18:33,387 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-09-15 11:18:33,387 - DEBUG - Project uuid in Contrail is c7452f82-ea73-460a-b135-7f161da7aa83
2025-09-15 11:18:33,388 - DEBUG - [10.0.0.19]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-09-15 11:18:33,514 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-09-15 11:18:33,514 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_NamespaceDatabaseList?namespace_uuid=c9ef97b3-813d-4cee-9458-277437afd791
2025-09-15 11:18:33,518 - INFO - Namespace default with uuid c9ef97b3-813d-4cee-9458-277437afd791 found in kube manager
2025-09-15 11:18:33,518 - INFO - Namespace default verification passed
2025-09-15 11:18:33,521 - DEBUG - Service ctest-nginx-svc-37866346 not present
2025-09-15 11:18:33,521 - INFO - Creating service ctest-nginx-svc-37866346
2025-09-15 11:18:33,535 - INFO - Created Service ctest-nginx-svc-37866346
2025-09-15 11:18:33,544 - DEBUG - Pod ctest-nginx-pod-95855735 not present
2025-09-15 11:18:33,545 - INFO - Creating Pod ctest-nginx-pod-95855735
2025-09-15 11:18:33,572 - DEBUG - Node for Pod ctest-nginx-pod-95855735 not yet populated
2025-09-15 11:18:36,580 - DEBUG - Pod : ctest-nginx-pod-95855735 UUID is 8ff726e0-cf16-43d4-9647-74cf66bc980f
2025-09-15 11:18:36,582 - DEBUG - Pod ctest-nginx-pod-65585122 not present
2025-09-15 11:18:36,582 - INFO - Creating Pod ctest-nginx-pod-65585122
2025-09-15 11:18:36,612 - DEBUG - Pod : ctest-nginx-pod-65585122 UUID is cc2dcd42-58ec-45bf-b1d0-5e0357d5a6bd
2025-09-15 11:18:36,782 - INFO - Created VN __public__, UUID :d3f4a5d6-76dc-4562-a0bb-dd241a726d76
2025-09-15 11:18:37,118 - DEBUG - Requesting: http://10.0.0.19:8082/domains
2025-09-15 11:18:37,129 - DEBUG - Requesting: http://10.0.0.19:8082/domain/bdfd9827-8d27-487e-a08d-2d9ad7182b3f
2025-09-15 11:18:37,141 - DEBUG - Requesting: http://10.0.0.19:8082/project/c7452f82-ea73-460a-b135-7f161da7aa83
2025-09-15 11:18:37,177 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/d3f4a5d6-76dc-4562-a0bb-dd241a726d76
2025-09-15 11:18:37,185 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/d3f4a5d6-76dc-4562-a0bb-dd241a726d76
2025-09-15 11:18:37,192 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/c73c7219-3815-46c6-bbe0-5685ad8f5ab4
2025-09-15 11:18:37,198 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/c73c7219-3815-46c6-bbe0-5685ad8f5ab4
2025-09-15 11:18:37,206 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/2386a164-999f-487d-97dc-68776ee252ef
2025-09-15 11:18:37,227 - DEBUG - Route Targets: ['target:64512:8000006']
2025-09-15 11:18:37,229 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/d3f4a5d6-76dc-4562-a0bb-dd241a726d76
2025-09-15 11:18:37,237 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/c73c7219-3815-46c6-bbe0-5685ad8f5ab4
2025-09-15 11:18:37,247 - INFO - Verified VN network id 13 for VN d3f4a5d6-76dc-4562-a0bb-dd241a726d76
2025-09-15 11:18:37,247 - INFO - Verifications in API Server for VN __public__ passed
2025-09-15 11:18:37,247 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/d3f4a5d6-76dc-4562-a0bb-dd241a726d76
2025-09-15 11:18:37,253 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/c73c7219-3815-46c6-bbe0-5685ad8f5ab4
2025-09-15 11:18:37,257 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/c73c7219-3815-46c6-bbe0-5685ad8f5ab4
2025-09-15 11:18:37,262 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/2386a164-999f-487d-97dc-68776ee252ef
2025-09-15 11:18:37,266 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-09-15 11:18:37,270 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15273014576855926114', 'uuid-lslong': '11582094013845630326'}, 'enable': 'true', 'created': '2025-09-15T11:18:36', 'last-modified': '2025-09-15T11:18:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.272228'}
2025-09-15 11:18:37,270 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 11:18:37,281 - DEBUG - Route Targets: ['target:64512:8000006']
2025-09-15 11:18:37,283 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-09-15 11:18:37,283 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-09-15 11:18:37,283 - DEBUG - Requesting: http://10.0.0.19:8082/domains
2025-09-15 11:18:37,294 - DEBUG - Requesting: http://10.0.0.19:8082/domain/bdfd9827-8d27-487e-a08d-2d9ad7182b3f
2025-09-15 11:18:37,300 - DEBUG - Requesting: http://10.0.0.19:8082/project/c7452f82-ea73-460a-b135-7f161da7aa83
2025-09-15 11:18:37,334 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/d3f4a5d6-76dc-4562-a0bb-dd241a726d76
2025-09-15 11:18:37,341 - DEBUG - =>VN __public__ has no policy to be verified
2025-09-15 11:18:37,341 - DEBUG - Verifying the vn in opserver
2025-09-15 11:18:37,341 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.19
2025-09-15 11:18:37,341 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks
2025-09-15 11:18:37,347 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-09-15 11:18:37,347 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-09-15 11:18:37,347 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 11:18:37,361 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 11:18:37,368 - DEBUG - Do not have enough data to verify VN in agent
2025-09-15 11:18:37,368 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 11:18:37,384 - DEBUG - VRF ids for VN __public__: {}
2025-09-15 11:18:37,469 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-09-15 11:18:37,559 - DEBUG - Requesting: http://10.0.0.19:8082/domains
2025-09-15 11:18:37,563 - DEBUG - Requesting: http://10.0.0.19:8082/domain/bdfd9827-8d27-487e-a08d-2d9ad7182b3f
2025-09-15 11:18:37,569 - DEBUG - Requesting: http://10.0.0.19:8082/project/c7452f82-ea73-460a-b135-7f161da7aa83
2025-09-15 11:18:37,599 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/d3f4a5d6-76dc-4562-a0bb-dd241a726d76
2025-09-15 11:18:37,612 - DEBUG - Requesting: http://10.0.0.19:8082/floating-ip-pool/ebc5d3c8-eda4-4a51-a954-11d38f6b01a4
2025-09-15 11:18:37,617 - DEBUG - Requesting: http://10.0.0.19:8082/domains
2025-09-15 11:18:37,621 - DEBUG - Requesting: http://10.0.0.19:8082/domain/bdfd9827-8d27-487e-a08d-2d9ad7182b3f
2025-09-15 11:18:37,627 - DEBUG - Requesting: http://10.0.0.19:8082/project/c7452f82-ea73-460a-b135-7f161da7aa83
2025-09-15 11:18:37,661 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/d3f4a5d6-76dc-4562-a0bb-dd241a726d76
2025-09-15 11:18:37,672 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-09-15 11:18:37,672 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-09-15 11:18:37,679 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-09-15 11:18:37,683 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-09-15 11:18:37,683 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-09-15 11:18:37,685 - DEBUG - Ingress ctest-nginx-ingress-74064184 not present
2025-09-15 11:18:37,686 - INFO - Creating Ingress ctest-nginx-ingress-74064184
2025-09-15 11:18:37,692 - INFO - Created Ingress ctest-nginx-ingress-74064184
2025-09-15 11:18:37,695 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-74064184
2025-09-15 11:18:40,700 - DEBUG - For Ingress ctest-nginx-ingress-74064184, Cluster IP: 10.47.255.249, External IPs ['5.17.125.60']
2025-09-15 11:18:40,700 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-74064184
2025-09-15 11:18:40,700 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_LoadbalancerDatabaseList?x=551bc2d7-3585-47b8-96b0-a2d345b644cc
2025-09-15 11:18:40,704 - INFO - Ingress ctest-nginx-ingress-74064184 with uuid 551bc2d7-3585-47b8-96b0-a2d345b644cc found in kube manager
2025-09-15 11:18:40,704 - INFO - Ingress ctest-nginx-ingress-74064184 verification passed
2025-09-15 11:18:40,706 - DEBUG - Pod ctest-busybox-pod-64459258 not present
2025-09-15 11:18:40,706 - INFO - Creating Pod ctest-busybox-pod-64459258
2025-09-15 11:18:40,744 - DEBUG - Pod : ctest-busybox-pod-64459258 UUID is 6afee4c4-60ca-4b25-ba2b-e089b6d133c5
2025-09-15 11:18:40,755 - DEBUG - Pod ctest-nginx-pod-95855735 not in running state.Currently in Pending
2025-09-15 11:18:45,761 - DEBUG - Pod ctest-nginx-pod-95855735 not in running state.Currently in Pending
2025-09-15 11:18:50,769 - INFO - Pod ctest-nginx-pod-95855735 is in running state.Got IP 10.47.255.251
2025-09-15 11:18:50,780 - DEBUG - Pod ctest-nginx-pod-95855735 has vmi b826a9aa-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:18:50,780 - INFO - Verified pod ctest-nginx-pod-95855735 in contrail-api
2025-09-15 11:18:50,780 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:18:50,792 - DEBUG - VMI b826a9aa-9225-11f0-8265-fa163ea53c0d is active in agent 10.0.0.137
2025-09-15 11:18:50,792 - INFO - Verified Pod ctest-nginx-pod-95855735 in agent 10.0.0.137
2025-09-15 11:18:50,793 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_VirtualMachineDatabaseList?x=8ff726e0-cf16-43d4-9647-74cf66bc980f
2025-09-15 11:18:50,796 - INFO - Pod ctest-nginx-pod-95855735 with uuid 8ff726e0-cf16-43d4-9647-74cf66bc980f found in kube manager
2025-09-15 11:18:50,796 - INFO - Pod ctest-nginx-pod-95855735 verification passed
2025-09-15 11:18:50,868 - DEBUG - [Pod ctest-nginx-pod-95855735] Cmd: echo ctest-nginx-pod-95855735 > /usr/share/nginx/html/index.html, Output:
2025-09-15 11:18:50,875 - INFO - Pod ctest-nginx-pod-65585122 is in running state.Got IP 10.47.255.250
2025-09-15 11:18:50,926 - DEBUG - Pod ctest-nginx-pod-65585122 has vmi b93b23de-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:18:50,926 - INFO - Verified pod ctest-nginx-pod-65585122 in contrail-api
2025-09-15 11:18:50,926 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:18:50,936 - DEBUG - VMI b93b23de-9225-11f0-8265-fa163ea53c0d is active in agent 10.0.0.137
2025-09-15 11:18:50,936 - INFO - Verified Pod ctest-nginx-pod-65585122 in agent 10.0.0.137
2025-09-15 11:18:50,936 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_VirtualMachineDatabaseList?x=cc2dcd42-58ec-45bf-b1d0-5e0357d5a6bd
2025-09-15 11:18:50,939 - INFO - Pod ctest-nginx-pod-65585122 with uuid cc2dcd42-58ec-45bf-b1d0-5e0357d5a6bd found in kube manager
2025-09-15 11:18:50,940 - INFO - Pod ctest-nginx-pod-65585122 verification passed
2025-09-15 11:18:51,010 - DEBUG - [Pod ctest-nginx-pod-65585122] Cmd: echo ctest-nginx-pod-65585122 > /usr/share/nginx/html/index.html, Output:
2025-09-15 11:18:51,016 - DEBUG - Pod ctest-busybox-pod-64459258 not in running state.Currently in Pending
2025-09-15 11:18:56,024 - INFO - Pod ctest-busybox-pod-64459258 is in running state.Got IP 10.47.255.248
2025-09-15 11:18:56,041 - DEBUG - Pod ctest-busybox-pod-64459258 has vmi bbb40f04-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:18:56,042 - INFO - Verified pod ctest-busybox-pod-64459258 in contrail-api
2025-09-15 11:18:56,042 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:18:56,051 - DEBUG - VMI bbb40f04-9225-11f0-8265-fa163ea53c0d is active in agent 10.0.0.137
2025-09-15 11:18:56,051 - INFO - Verified Pod ctest-busybox-pod-64459258 in agent 10.0.0.137
2025-09-15 11:18:56,051 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_VirtualMachineDatabaseList?x=6afee4c4-60ca-4b25-ba2b-e089b6d133c5
2025-09-15 11:18:56,055 - INFO - Pod ctest-busybox-pod-64459258 with uuid 6afee4c4-60ca-4b25-ba2b-e089b6d133c5 found in kube manager
2025-09-15 11:18:56,055 - INFO - Pod ctest-busybox-pod-64459258 verification passed
2025-09-15 11:18:56,115 - DEBUG - [Pod ctest-busybox-pod-64459258] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-95855735
2025-09-15 11:18:56,115 - DEBUG - [Pod Pod ctest-busybox-pod-64459258] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-09-15 11:18:56,115 - DEBUG - [Pod Pod ctest-busybox-pod-64459258] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-95855735
2025-09-15 11:18:56,170 - DEBUG - [Pod ctest-busybox-pod-64459258] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-65585122
2025-09-15 11:18:56,170 - DEBUG - [Pod Pod ctest-busybox-pod-64459258] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-09-15 11:18:56,170 - DEBUG - [Pod Pod ctest-busybox-pod-64459258] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-65585122
2025-09-15 11:18:56,170 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-95855735': 1, 'ctest-nginx-pod-65585122': 1}
2025-09-15 11:18:56,170 - WARNING - Nothing to delete parallely
2025-09-15 11:18:56,171 - INFO - Deleting pod default:ctest-busybox-pod-64459258
2025-09-15 11:18:56,182 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=6afee4c4-60ca-4b25-ba2b-e089b6d133c5
2025-09-15 11:18:56,189 - WARNING - Pod uuid 6afee4c4-60ca-4b25-ba2b-e089b6d133c5 is still seen in agent 10.0.0.137 VM list
2025-09-15 11:19:01,189 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=6afee4c4-60ca-4b25-ba2b-e089b6d133c5
2025-09-15 11:19:01,196 - WARNING - Pod uuid 6afee4c4-60ca-4b25-ba2b-e089b6d133c5 is still seen in agent 10.0.0.137 VM list
2025-09-15 11:19:06,196 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=6afee4c4-60ca-4b25-ba2b-e089b6d133c5
2025-09-15 11:19:06,205 - WARNING - Pod uuid 6afee4c4-60ca-4b25-ba2b-e089b6d133c5 is still seen in agent 10.0.0.137 VM list
2025-09-15 11:19:11,206 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=6afee4c4-60ca-4b25-ba2b-e089b6d133c5
2025-09-15 11:19:11,213 - WARNING - Pod uuid 6afee4c4-60ca-4b25-ba2b-e089b6d133c5 is still seen in agent 10.0.0.137 VM list
2025-09-15 11:19:16,213 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=6afee4c4-60ca-4b25-ba2b-e089b6d133c5
2025-09-15 11:19:16,220 - WARNING - Pod uuid 6afee4c4-60ca-4b25-ba2b-e089b6d133c5 is still seen in agent 10.0.0.137 VM list
2025-09-15 11:19:21,221 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=6afee4c4-60ca-4b25-ba2b-e089b6d133c5
2025-09-15 11:19:21,231 - WARNING - Pod uuid 6afee4c4-60ca-4b25-ba2b-e089b6d133c5 is still seen in agent 10.0.0.137 VM list
2025-09-15 11:19:26,232 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=6afee4c4-60ca-4b25-ba2b-e089b6d133c5
2025-09-15 11:19:26,239 - WARNING - Pod uuid 6afee4c4-60ca-4b25-ba2b-e089b6d133c5 is still seen in agent 10.0.0.137 VM list
2025-09-15 11:19:31,240 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=6afee4c4-60ca-4b25-ba2b-e089b6d133c5
2025-09-15 11:19:31,246 - DEBUG - Pod 6afee4c4-60ca-4b25-ba2b-e089b6d133c5 is not in agent 10.0.0.137 VM list
2025-09-15 11:19:31,246 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:bbb40f04-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:19:31,251 - DEBUG - VMI bbb40f04-9225-11f0-8265-fa163ea53c0d is removed from agent 10.0.0.137
2025-09-15 11:19:31,251 - INFO - Verified that pod ctest-busybox-pod-64459258 is removed in agent
2025-09-15 11:19:31,251 - INFO - Deleting Ingress : ctest-nginx-ingress-74064184
2025-09-15 11:19:31,270 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_LoadbalancerDatabaseList?x=551bc2d7-3585-47b8-96b0-a2d345b644cc
2025-09-15 11:19:31,274 - ERROR - Ingress ctest-nginx-ingress-74064184 with uuid 551bc2d7-3585-47b8-96b0-a2d345b644cc still found in kube manager
2025-09-15 11:19:32,274 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_LoadbalancerDatabaseList?x=551bc2d7-3585-47b8-96b0-a2d345b644cc
2025-09-15 11:19:32,278 - INFO - Ingress ctest-nginx-ingress-74064184 with uuid 551bc2d7-3585-47b8-96b0-a2d345b644cc deleted successfully from kube manager
2025-09-15 11:19:32,278 - INFO - Verifications on Ingress %s cleanup passed
2025-09-15 11:19:32,278 - INFO - Deleting the FIP pool __fip_pool_public__
2025-09-15 11:19:32,355 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-09-15 11:19:32,360 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-09-15 11:19:32,360 - INFO - Deleting VN __public__
2025-09-15 11:19:32,494 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/c73c7219-3815-46c6-bbe0-5685ad8f5ab4
2025-09-15 11:19:32,503 - DEBUG - Response Code: 404
2025-09-15 11:19:32,503 - DEBUG - Requesting: http://10.0.0.19:8082/domains
2025-09-15 11:19:32,509 - DEBUG - Requesting: http://10.0.0.19:8082/domain/bdfd9827-8d27-487e-a08d-2d9ad7182b3f
2025-09-15 11:19:32,519 - DEBUG - Requesting: http://10.0.0.19:8082/project/c7452f82-ea73-460a-b135-7f161da7aa83
2025-09-15 11:19:32,581 - INFO - Validated that VN __public__ is not found in API Server
2025-09-15 11:19:32,581 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-09-15 11:19:32,591 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 11:19:32,597 - DEBUG - VN __public__ is not present in Agent 10.0.0.137
2025-09-15 11:19:32,597 - INFO - Validated that VN __public__ is not in any agent
2025-09-15 11:19:32,598 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 11:19:32,605 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 11:19:32,610 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-09-15 11:19:32,614 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-09-15 11:19:32,615 - INFO - Deleting pod default:ctest-nginx-pod-65585122
2025-09-15 11:19:32,637 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=cc2dcd42-58ec-45bf-b1d0-5e0357d5a6bd
2025-09-15 11:19:32,649 - WARNING - Pod uuid cc2dcd42-58ec-45bf-b1d0-5e0357d5a6bd is still seen in agent 10.0.0.137 VM list
2025-09-15 11:19:37,650 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=cc2dcd42-58ec-45bf-b1d0-5e0357d5a6bd
2025-09-15 11:19:37,656 - DEBUG - Pod cc2dcd42-58ec-45bf-b1d0-5e0357d5a6bd is not in agent 10.0.0.137 VM list
2025-09-15 11:19:37,656 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b93b23de-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:19:37,660 - DEBUG - VMI b93b23de-9225-11f0-8265-fa163ea53c0d is removed from agent 10.0.0.137
2025-09-15 11:19:37,661 - INFO - Verified that pod ctest-nginx-pod-65585122 is removed in agent
2025-09-15 11:19:37,661 - INFO - Deleting pod default:ctest-nginx-pod-95855735
2025-09-15 11:19:37,682 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=8ff726e0-cf16-43d4-9647-74cf66bc980f
2025-09-15 11:19:37,697 - WARNING - Pod uuid 8ff726e0-cf16-43d4-9647-74cf66bc980f is still seen in agent 10.0.0.137 VM list
2025-09-15 11:19:42,697 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=8ff726e0-cf16-43d4-9647-74cf66bc980f
2025-09-15 11:19:42,703 - DEBUG - Pod 8ff726e0-cf16-43d4-9647-74cf66bc980f is not in agent 10.0.0.137 VM list
2025-09-15 11:19:42,703 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b826a9aa-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:19:42,708 - DEBUG - VMI b826a9aa-9225-11f0-8265-fa163ea53c0d is removed from agent 10.0.0.137
2025-09-15 11:19:42,708 - INFO - Verified that pod ctest-nginx-pod-95855735 is removed in agent
2025-09-15 11:19:42,708 - INFO - Deleting service : ctest-nginx-svc-37866346
2025-09-15 11:19:42,859 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_AgentXmppConnectionStatusReq?
2025-09-15 11:19:42,865 - DEBUG - No XMPP flaps were noticed during the test
2025-09-15 11:19:42,865 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:09]
2025-09-15 11:19:42,865 - INFO - --------------------------------------------------------------------------------
2025-09-15 11:19:49,196 - DEBUG - [10.0.0.19]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-09-15 11:19:49,318 - DEBUG - Output : cluster_project={}
2025-09-15 11:19:49,361 - DEBUG - [10.0.0.19]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-09-15 11:19:49,532 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-09-15 11:19:49,537 - DEBUG - [10.0.0.19]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-09-15 11:19:49,665 - DEBUG - Output : cluster_project={}
2025-09-15 11:20:04,751 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:20:04,787 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:20:04,860 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:20:09,902 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:20:09,980 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:20:10,054 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:20:10,085 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:20:10,160 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 11:20:10,314 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_AgentXmppConnectionStatusReq?
2025-09-15 11:20:12,070 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_ApplicationPolicySetReq?
2025-09-15 11:20:12,078 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_AclReq?
2025-09-15 11:20:12,113 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_AclReq?
2025-09-15 11:20:14,137 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_AclReq?
2025-09-15 11:23:20,970 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_AgentXmppConnectionStatusReq?
2025-09-15 11:23:21,013 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=01554500-47aa-488b-bdc5-7461f156f2f5
2025-09-15 11:23:21,019 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=4f017daf-93c6-4825-9729-1e9b4e009a43
2025-09-15 11:23:21,026 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=81f5cab4-eda1-4253-8912-bd6002b34090
2025-09-15 11:23:21,032 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=e054fd21-07d8-464f-9680-e7f7df8dbf01
2025-09-15 11:23:21,038 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=c5a49c00-ca61-4210-9609-cc260026e093
2025-09-15 11:23:21,049 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=41772bff-b394-40c6-8d27-2a79cf1b9159
2025-09-15 11:23:21,053 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=49368d58-eae7-421f-ae77-6226a93b8b94
2025-09-15 11:23:21,058 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=4975291d-8bdc-453f-bb2c-b2f49f89d623
2025-09-15 11:23:26,062 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=4f017daf-93c6-4825-9729-1e9b4e009a43
2025-09-15 11:23:26,063 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=01554500-47aa-488b-bdc5-7461f156f2f5
2025-09-15 11:23:26,065 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=81f5cab4-eda1-4253-8912-bd6002b34090
2025-09-15 11:23:26,074 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e5b712ba-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:23:26,076 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e653cbaa-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:23:26,077 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e53f2944-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:23:26,115 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=e054fd21-07d8-464f-9680-e7f7df8dbf01
2025-09-15 11:23:26,117 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=c5a49c00-ca61-4210-9609-cc260026e093
2025-09-15 11:23:26,118 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=4975291d-8bdc-453f-bb2c-b2f49f89d623
2025-09-15 11:23:26,119 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=49368d58-eae7-421f-ae77-6226a93b8b94
2025-09-15 11:23:26,120 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=41772bff-b394-40c6-8d27-2a79cf1b9159
2025-09-15 11:23:26,133 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e6c14d38-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:23:31,135 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=e054fd21-07d8-464f-9680-e7f7df8dbf01
2025-09-15 11:23:31,136 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=4975291d-8bdc-453f-bb2c-b2f49f89d623
2025-09-15 11:23:31,137 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=49368d58-eae7-421f-ae77-6226a93b8b94
2025-09-15 11:23:31,138 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=41772bff-b394-40c6-8d27-2a79cf1b9159
2025-09-15 11:23:36,151 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=e054fd21-07d8-464f-9680-e7f7df8dbf01
2025-09-15 11:23:36,153 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=4975291d-8bdc-453f-bb2c-b2f49f89d623
2025-09-15 11:23:36,154 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=49368d58-eae7-421f-ae77-6226a93b8b94
2025-09-15 11:23:36,159 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=41772bff-b394-40c6-8d27-2a79cf1b9159
2025-09-15 11:23:41,170 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=e054fd21-07d8-464f-9680-e7f7df8dbf01
2025-09-15 11:23:41,172 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=49368d58-eae7-421f-ae77-6226a93b8b94
2025-09-15 11:23:41,173 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=41772bff-b394-40c6-8d27-2a79cf1b9159
2025-09-15 11:23:41,174 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=4975291d-8bdc-453f-bb2c-b2f49f89d623
2025-09-15 11:23:46,185 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=4975291d-8bdc-453f-bb2c-b2f49f89d623
2025-09-15 11:23:46,186 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=e054fd21-07d8-464f-9680-e7f7df8dbf01
2025-09-15 11:23:46,187 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=41772bff-b394-40c6-8d27-2a79cf1b9159
2025-09-15 11:23:46,188 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=49368d58-eae7-421f-ae77-6226a93b8b94
2025-09-15 11:23:51,199 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=e054fd21-07d8-464f-9680-e7f7df8dbf01
2025-09-15 11:23:51,201 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=4975291d-8bdc-453f-bb2c-b2f49f89d623
2025-09-15 11:23:51,202 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=41772bff-b394-40c6-8d27-2a79cf1b9159
2025-09-15 11:23:51,203 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=49368d58-eae7-421f-ae77-6226a93b8b94
2025-09-15 11:23:56,218 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=4975291d-8bdc-453f-bb2c-b2f49f89d623
2025-09-15 11:23:56,219 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=e054fd21-07d8-464f-9680-e7f7df8dbf01
2025-09-15 11:23:56,220 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=41772bff-b394-40c6-8d27-2a79cf1b9159
2025-09-15 11:23:56,222 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_VmListReq?uuid=49368d58-eae7-421f-ae77-6226a93b8b94
2025-09-15 11:23:56,234 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e8772fb2-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:23:56,235 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e81a4c3e-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:23:56,237 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e739df50-9225-11f0-8265-fa163ea53c0d
2025-09-15 11:23:56,238 - DEBUG - Requesting: http://10.0.0.137:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e79d4e28-9225-11f0-8265-fa163ea53c0d