2025-10-31 00:35:01,230 - DEBUG - [10.0.0.61]: Running cmd : hostname
2025-10-31 00:35:01,506 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1662-1
2025-10-31 00:35:01,506 - DEBUG - [10.0.0.61]: Running cmd : hostname -f
2025-10-31 00:35:01,540 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1662-1
2025-10-31 00:35:01,540 - DEBUG - [10.0.0.61]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-10-31 00:35:01,646 - DEBUG - Output : NAMES
contrail_test_xbzacHiGt
contrail-kubernetes-master-kubemanager-1
webui-web-1
webui-job-1
analytics_database-cassandra-1
analytics_database-nodemgr-1
analytics_database-query-engine-1
analytics_snmp-snmp-collector-1
analytics_snmp-topology-1
analytics_snmp-nodemgr-1
analytics_alarm-kafka-1
analytics_alarm-alarm-gen-1
analytics_alarm-nodemgr-1
analytics-collector-1
analytics-api-1
analytics-nodemgr-1
redis-redis-1
control-named-1
control-nodemgr-1
control-control-1
control-dns-1
config_database-nodemgr-1
config_database-cassandra-1
config_database-rabbitmq-1
config_database-zookeeper-1
config_api-devicemgr-1
config_api-api-1
config_api-nodemgr-1
config_api-svcmonitor-1
config_api-dnsmasq-1
config_api-schema-1
2025-10-31 00:35:01,646 - DEBUG - [10.0.0.61]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-10-31 00:35:01,681 - DEBUG - Output : 127.0.0.1/8
10.0.0.61/24
252.61.0.1/8
10.159.252.1/24
172.17.0.1/16
2025-10-31 00:35:01,682 - DEBUG - [10.0.0.61]: Running cmd : getent hosts 10.0.0.61 | head -n 1 | awk '{print $2}'
2025-10-31 00:35:01,716 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1662-1
2025-10-31 00:35:01,830 - DEBUG - [10.0.0.40]: Running cmd : hostname
2025-10-31 00:35:02,046 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1662-1
2025-10-31 00:35:02,047 - DEBUG - [10.0.0.40]: Running cmd : hostname -f
2025-10-31 00:35:02,120 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1662-1
2025-10-31 00:35:02,120 - DEBUG - [10.0.0.40]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-10-31 00:35:02,165 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2025-10-31 00:35:02,165 - DEBUG - [10.0.0.40]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-10-31 00:35:02,270 - DEBUG - Output : 10.0.0.40/24
2025-10-31 00:35:02,270 - DEBUG - [10.0.0.40]: Running cmd : getent hosts 10.0.0.40 | head -n 1 | awk '{print $2}'
2025-10-31 00:35:02,304 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1662-1
2025-10-31 00:35:02,369 - INFO - Using existing project ['default-domain', 'k8s-default'](e1aa05eb-4bfb-4f92-ac9d-9c7f2e3c2aa3)
2025-10-31 00:35:02,430 - INFO - Using existing project ['default-domain', 'k8s-default'](e1aa05eb-4bfb-4f92-ac9d-9c7f2e3c2aa3)
2025-10-31 00:35:02,543 - INFO - ================================================================================
2025-10-31 00:35:02,543 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-10-31 00:35:02,543 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-10-31 00:35:02,796 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AgentXmppConnectionStatusReq?
2025-10-31 00:35:02,804 - DEBUG - Nothing to compare xmpp stats {'10.0.0.40': {'10.0.0.61': '0'}} with
2025-10-31 00:35:02,804 - INFO - Initial checks done. Running the testcase now
2025-10-31 00:35:02,805 - INFO -
2025-10-31 00:35:02,809 - INFO - Namespace default already exists
2025-10-31 00:35:02,809 - DEBUG - [10.0.0.61]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-10-31 00:35:02,936 - DEBUG - Output : cluster_project={}
2025-10-31 00:35:03,015 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-10-31 00:35:03,015 - DEBUG - Project uuid in Contrail is e1aa05eb-4bfb-4f92-ac9d-9c7f2e3c2aa3
2025-10-31 00:35:03,015 - DEBUG - [10.0.0.61]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-10-31 00:35:03,152 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-10-31 00:35:03,152 - DEBUG - Requesting: http://10.0.0.61:8108/Snh_NamespaceDatabaseList?namespace_uuid=cf7232ea-d75d-45c0-bc2b-ad15a51fa72a
2025-10-31 00:35:03,157 - INFO - Namespace default with uuid cf7232ea-d75d-45c0-bc2b-ad15a51fa72a found in kube manager
2025-10-31 00:35:03,157 - INFO - Namespace default verification passed
2025-10-31 00:35:03,160 - DEBUG - Service ctest-nginx-svc-40951002 not present
2025-10-31 00:35:03,160 - INFO - Creating service ctest-nginx-svc-40951002
2025-10-31 00:35:03,188 - INFO - Created Service ctest-nginx-svc-40951002
2025-10-31 00:35:03,191 - DEBUG - Pod ctest-nginx-pod-18649308 not present
2025-10-31 00:35:03,191 - INFO - Creating Pod ctest-nginx-pod-18649308
2025-10-31 00:35:03,257 - DEBUG - Node for Pod ctest-nginx-pod-18649308 not yet populated
2025-10-31 00:35:06,266 - DEBUG - Pod : ctest-nginx-pod-18649308 UUID is 83ef1ec7-cb81-4eba-bf9f-d182180dd961
2025-10-31 00:35:06,269 - DEBUG - Pod ctest-nginx-pod-94011549 not present
2025-10-31 00:35:06,269 - INFO - Creating Pod ctest-nginx-pod-94011549
2025-10-31 00:35:06,307 - DEBUG - Pod : ctest-nginx-pod-94011549 UUID is dd882b0e-4924-4009-9759-3c914411dac8
2025-10-31 00:35:06,531 - INFO - Created VN __public__, UUID :8a618029-20ab-4559-80cb-12a441a51032
2025-10-31 00:35:06,755 - DEBUG - Requesting: http://10.0.0.61:8082/domains
2025-10-31 00:35:06,761 - DEBUG - Requesting: http://10.0.0.61:8082/domain/baeb4ece-e0da-438d-906b-54b1de16acc9
2025-10-31 00:35:06,774 - DEBUG - Requesting: http://10.0.0.61:8082/project/e1aa05eb-4bfb-4f92-ac9d-9c7f2e3c2aa3
2025-10-31 00:35:06,836 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/8a618029-20ab-4559-80cb-12a441a51032
2025-10-31 00:35:06,873 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/8a618029-20ab-4559-80cb-12a441a51032
2025-10-31 00:35:06,882 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/c57bbe29-d80b-456e-a825-c6bccb64f2f8
2025-10-31 00:35:06,888 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/c57bbe29-d80b-456e-a825-c6bccb64f2f8
2025-10-31 00:35:06,908 - DEBUG - Requesting: http://10.0.0.61:8082/route-target/ec2fe22c-5bd4-416e-be7d-24862e477d3f
2025-10-31 00:35:06,914 - DEBUG - Route Targets: ['target:64512:8000006']
2025-10-31 00:35:06,914 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/8a618029-20ab-4559-80cb-12a441a51032
2025-10-31 00:35:06,922 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/c57bbe29-d80b-456e-a825-c6bccb64f2f8
2025-10-31 00:35:06,947 - INFO - Verified VN network id 13 for VN 8a618029-20ab-4559-80cb-12a441a51032
2025-10-31 00:35:06,947 - INFO - Verifications in API Server for VN __public__ passed
2025-10-31 00:35:06,947 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/8a618029-20ab-4559-80cb-12a441a51032
2025-10-31 00:35:06,956 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/c57bbe29-d80b-456e-a825-c6bccb64f2f8
2025-10-31 00:35:06,964 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/c57bbe29-d80b-456e-a825-c6bccb64f2f8
2025-10-31 00:35:06,975 - DEBUG - Requesting: http://10.0.0.61:8082/route-target/ec2fe22c-5bd4-416e-be7d-24862e477d3f
2025-10-31 00:35:06,981 - DEBUG - Requesting: http://10.0.0.61:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-10-31 00:35:07,002 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9971391964105098585', 'uuid-lslong': '9280531953812312114'}, 'enable': 'true', 'created': '2025-10-31T00:35:06', 'last-modified': '2025-10-31T00:35:06', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.370425'}
2025-10-31 00:35:07,002 - DEBUG - Requesting: http://10.0.0.61:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-10-31 00:35:07,021 - DEBUG - Route Targets: ['target:64512:8000006']
2025-10-31 00:35:07,021 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-10-31 00:35:07,021 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-10-31 00:35:07,022 - DEBUG - Requesting: http://10.0.0.61:8082/domains
2025-10-31 00:35:07,026 - DEBUG - Requesting: http://10.0.0.61:8082/domain/baeb4ece-e0da-438d-906b-54b1de16acc9
2025-10-31 00:35:07,036 - DEBUG - Requesting: http://10.0.0.61:8082/project/e1aa05eb-4bfb-4f92-ac9d-9c7f2e3c2aa3
2025-10-31 00:35:07,081 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/8a618029-20ab-4559-80cb-12a441a51032
2025-10-31 00:35:07,093 - DEBUG - =>VN __public__ has no policy to be verified
2025-10-31 00:35:07,093 - DEBUG - Verifying the vn in opserver
2025-10-31 00:35:07,093 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.61
2025-10-31 00:35:07,093 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks
2025-10-31 00:35:07,097 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-10-31 00:35:07,097 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-10-31 00:35:07,097 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-10-31 00:35:07,107 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-10-31 00:35:07,117 - DEBUG - Do not have enough data to verify VN in agent
2025-10-31 00:35:07,117 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-10-31 00:35:07,126 - DEBUG - VRF ids for VN __public__: {}
2025-10-31 00:35:07,227 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-10-31 00:35:07,329 - DEBUG - Requesting: http://10.0.0.61:8082/domains
2025-10-31 00:35:07,335 - DEBUG - Requesting: http://10.0.0.61:8082/domain/baeb4ece-e0da-438d-906b-54b1de16acc9
2025-10-31 00:35:07,342 - DEBUG - Requesting: http://10.0.0.61:8082/project/e1aa05eb-4bfb-4f92-ac9d-9c7f2e3c2aa3
2025-10-31 00:35:07,394 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/8a618029-20ab-4559-80cb-12a441a51032
2025-10-31 00:35:07,407 - DEBUG - Requesting: http://10.0.0.61:8082/floating-ip-pool/5a79419c-0f3f-4fbf-86f2-a503d8f0aa70
2025-10-31 00:35:07,412 - DEBUG - Requesting: http://10.0.0.61:8082/domains
2025-10-31 00:35:07,416 - DEBUG - Requesting: http://10.0.0.61:8082/domain/baeb4ece-e0da-438d-906b-54b1de16acc9
2025-10-31 00:35:07,423 - DEBUG - Requesting: http://10.0.0.61:8082/project/e1aa05eb-4bfb-4f92-ac9d-9c7f2e3c2aa3
2025-10-31 00:35:07,460 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/8a618029-20ab-4559-80cb-12a441a51032
2025-10-31 00:35:07,468 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-10-31 00:35:07,468 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-10-31 00:35:07,474 - DEBUG - Requesting: http://10.0.0.61:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-10-31 00:35:07,480 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-10-31 00:35:07,480 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-10-31 00:35:07,483 - DEBUG - Ingress ctest-nginx-ingress-47288044 not present
2025-10-31 00:35:07,484 - INFO - Creating Ingress ctest-nginx-ingress-47288044
2025-10-31 00:35:07,492 - INFO - Created Ingress ctest-nginx-ingress-47288044
2025-10-31 00:35:07,497 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-47288044
2025-10-31 00:35:10,504 - DEBUG - For Ingress ctest-nginx-ingress-47288044, Cluster IP: 10.47.255.249, External IPs ['147.62.248.252']
2025-10-31 00:35:10,504 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-47288044
2025-10-31 00:35:10,504 - DEBUG - Requesting: http://10.0.0.61:8108/Snh_LoadbalancerDatabaseList?x=b1c4575d-8d1b-4382-96fc-ba5522e02338
2025-10-31 00:35:10,511 - INFO - Ingress ctest-nginx-ingress-47288044 with uuid b1c4575d-8d1b-4382-96fc-ba5522e02338 found in kube manager
2025-10-31 00:35:10,511 - INFO - Ingress ctest-nginx-ingress-47288044 verification passed
2025-10-31 00:35:10,515 - DEBUG - Pod ctest-busybox-pod-62471299 not present
2025-10-31 00:35:10,517 - INFO - Creating Pod ctest-busybox-pod-62471299
2025-10-31 00:35:10,549 - DEBUG - Pod : ctest-busybox-pod-62471299 UUID is bfdeebc1-b895-4d1b-8fd9-58cd02fd814d
2025-10-31 00:35:10,560 - DEBUG - Pod ctest-nginx-pod-18649308 not in running state.Currently in Pending
2025-10-31 00:35:15,568 - DEBUG - Pod ctest-nginx-pod-18649308 not in running state.Currently in Pending
2025-10-31 00:35:20,575 - DEBUG - Pod ctest-nginx-pod-18649308 not in running state.Currently in Pending
2025-10-31 00:35:25,582 - INFO - Pod ctest-nginx-pod-18649308 is in running state.Got IP 10.47.255.251
2025-10-31 00:35:25,643 - DEBUG - Pod ctest-nginx-pod-18649308 has vmi 71a53434-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:35:25,643 - INFO - Verified pod ctest-nginx-pod-18649308 in contrail-api
2025-10-31 00:35:25,643 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:35:25,654 - DEBUG - VMI 71a53434-b5f1-11f0-8589-fa163efc4e20 is active in agent 10.0.0.40
2025-10-31 00:35:25,654 - INFO - Verified Pod ctest-nginx-pod-18649308 in agent 10.0.0.40
2025-10-31 00:35:25,654 - DEBUG - Requesting: http://10.0.0.61:8108/Snh_VirtualMachineDatabaseList?x=83ef1ec7-cb81-4eba-bf9f-d182180dd961
2025-10-31 00:35:25,658 - INFO - Pod ctest-nginx-pod-18649308 with uuid 83ef1ec7-cb81-4eba-bf9f-d182180dd961 found in kube manager
2025-10-31 00:35:25,658 - INFO - Pod ctest-nginx-pod-18649308 verification passed
2025-10-31 00:35:25,734 - DEBUG - [Pod ctest-nginx-pod-18649308] Cmd: echo ctest-nginx-pod-18649308 > /usr/share/nginx/html/index.html, Output:
2025-10-31 00:35:25,741 - INFO - Pod ctest-nginx-pod-94011549 is in running state.Got IP 10.47.255.250
2025-10-31 00:35:25,760 - DEBUG - Pod ctest-nginx-pod-94011549 has vmi 72b82dea-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:35:25,760 - INFO - Verified pod ctest-nginx-pod-94011549 in contrail-api
2025-10-31 00:35:25,760 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:35:25,770 - DEBUG - VMI 72b82dea-b5f1-11f0-8589-fa163efc4e20 is active in agent 10.0.0.40
2025-10-31 00:35:25,770 - INFO - Verified Pod ctest-nginx-pod-94011549 in agent 10.0.0.40
2025-10-31 00:35:25,771 - DEBUG - Requesting: http://10.0.0.61:8108/Snh_VirtualMachineDatabaseList?x=dd882b0e-4924-4009-9759-3c914411dac8
2025-10-31 00:35:25,774 - INFO - Pod ctest-nginx-pod-94011549 with uuid dd882b0e-4924-4009-9759-3c914411dac8 found in kube manager
2025-10-31 00:35:25,774 - INFO - Pod ctest-nginx-pod-94011549 verification passed
2025-10-31 00:35:25,835 - DEBUG - [Pod ctest-nginx-pod-94011549] Cmd: echo ctest-nginx-pod-94011549 > /usr/share/nginx/html/index.html, Output:
2025-10-31 00:35:25,843 - INFO - Pod ctest-busybox-pod-62471299 is in running state.Got IP 10.47.255.248
2025-10-31 00:35:25,855 - DEBUG - Pod ctest-busybox-pod-62471299 has vmi 754cb6f2-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:35:25,855 - INFO - Verified pod ctest-busybox-pod-62471299 in contrail-api
2025-10-31 00:35:25,855 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:35:25,865 - DEBUG - VMI 754cb6f2-b5f1-11f0-8589-fa163efc4e20 is active in agent 10.0.0.40
2025-10-31 00:35:25,865 - INFO - Verified Pod ctest-busybox-pod-62471299 in agent 10.0.0.40
2025-10-31 00:35:25,865 - DEBUG - Requesting: http://10.0.0.61:8108/Snh_VirtualMachineDatabaseList?x=bfdeebc1-b895-4d1b-8fd9-58cd02fd814d
2025-10-31 00:35:25,869 - INFO - Pod ctest-busybox-pod-62471299 with uuid bfdeebc1-b895-4d1b-8fd9-58cd02fd814d found in kube manager
2025-10-31 00:35:25,869 - INFO - Pod ctest-busybox-pod-62471299 verification passed
2025-10-31 00:35:25,927 - DEBUG - [Pod ctest-busybox-pod-62471299] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-18649308
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-10-31 00:35:25,927 - DEBUG - [Pod Pod ctest-busybox-pod-62471299] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-10-31 00:35:25,928 - DEBUG - [Pod Pod ctest-busybox-pod-62471299] Cmd output: ctest-nginx-pod-18649308
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-10-31 00:35:25,985 - DEBUG - [Pod ctest-busybox-pod-62471299] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-94011549
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-10-31 00:35:25,985 - DEBUG - [Pod Pod ctest-busybox-pod-62471299] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-10-31 00:35:25,985 - DEBUG - [Pod Pod ctest-busybox-pod-62471299] Cmd output: ctest-nginx-pod-94011549
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-10-31 00:35:25,985 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-18649308': 1, 'ctest-nginx-pod-94011549': 1}
2025-10-31 00:35:25,985 - WARNING - Nothing to delete parallely
2025-10-31 00:35:25,986 - INFO - Deleting pod default:ctest-busybox-pod-62471299
2025-10-31 00:35:25,997 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=bfdeebc1-b895-4d1b-8fd9-58cd02fd814d
2025-10-31 00:35:26,006 - WARNING - Pod uuid bfdeebc1-b895-4d1b-8fd9-58cd02fd814d is still seen in agent 10.0.0.40 VM list
2025-10-31 00:35:31,006 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=bfdeebc1-b895-4d1b-8fd9-58cd02fd814d
2025-10-31 00:35:31,012 - WARNING - Pod uuid bfdeebc1-b895-4d1b-8fd9-58cd02fd814d is still seen in agent 10.0.0.40 VM list
2025-10-31 00:35:36,013 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=bfdeebc1-b895-4d1b-8fd9-58cd02fd814d
2025-10-31 00:35:36,022 - WARNING - Pod uuid bfdeebc1-b895-4d1b-8fd9-58cd02fd814d is still seen in agent 10.0.0.40 VM list
2025-10-31 00:35:41,023 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=bfdeebc1-b895-4d1b-8fd9-58cd02fd814d
2025-10-31 00:35:41,031 - WARNING - Pod uuid bfdeebc1-b895-4d1b-8fd9-58cd02fd814d is still seen in agent 10.0.0.40 VM list
2025-10-31 00:35:46,032 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=bfdeebc1-b895-4d1b-8fd9-58cd02fd814d
2025-10-31 00:35:46,040 - WARNING - Pod uuid bfdeebc1-b895-4d1b-8fd9-58cd02fd814d is still seen in agent 10.0.0.40 VM list
2025-10-31 00:35:51,041 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=bfdeebc1-b895-4d1b-8fd9-58cd02fd814d
2025-10-31 00:35:51,048 - WARNING - Pod uuid bfdeebc1-b895-4d1b-8fd9-58cd02fd814d is still seen in agent 10.0.0.40 VM list
2025-10-31 00:35:56,049 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=bfdeebc1-b895-4d1b-8fd9-58cd02fd814d
2025-10-31 00:35:56,058 - WARNING - Pod uuid bfdeebc1-b895-4d1b-8fd9-58cd02fd814d is still seen in agent 10.0.0.40 VM list
2025-10-31 00:36:01,060 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=bfdeebc1-b895-4d1b-8fd9-58cd02fd814d
2025-10-31 00:36:01,067 - DEBUG - Pod bfdeebc1-b895-4d1b-8fd9-58cd02fd814d is not in agent 10.0.0.40 VM list
2025-10-31 00:36:01,067 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:754cb6f2-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:36:01,072 - DEBUG - VMI 754cb6f2-b5f1-11f0-8589-fa163efc4e20 is removed from agent 10.0.0.40
2025-10-31 00:36:01,072 - INFO - Verified that pod ctest-busybox-pod-62471299 is removed in agent
2025-10-31 00:36:01,072 - INFO - Deleting Ingress : ctest-nginx-ingress-47288044
2025-10-31 00:36:01,079 - DEBUG - Requesting: http://10.0.0.61:8108/Snh_LoadbalancerDatabaseList?x=b1c4575d-8d1b-4382-96fc-ba5522e02338
2025-10-31 00:36:01,083 - ERROR - Ingress ctest-nginx-ingress-47288044 with uuid b1c4575d-8d1b-4382-96fc-ba5522e02338 still found in kube manager
2025-10-31 00:36:02,083 - DEBUG - Requesting: http://10.0.0.61:8108/Snh_LoadbalancerDatabaseList?x=b1c4575d-8d1b-4382-96fc-ba5522e02338
2025-10-31 00:36:02,087 - INFO - Ingress ctest-nginx-ingress-47288044 with uuid b1c4575d-8d1b-4382-96fc-ba5522e02338 deleted successfully from kube manager
2025-10-31 00:36:02,087 - INFO - Verifications on Ingress %s cleanup passed
2025-10-31 00:36:02,088 - INFO - Deleting the FIP pool __fip_pool_public__
2025-10-31 00:36:02,286 - DEBUG - Requesting: http://10.0.0.61:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-10-31 00:36:02,292 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-10-31 00:36:02,292 - INFO - Deleting VN __public__
2025-10-31 00:36:02,478 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/c57bbe29-d80b-456e-a825-c6bccb64f2f8
2025-10-31 00:36:02,484 - DEBUG - Response Code: 404
2025-10-31 00:36:02,485 - DEBUG - Requesting: http://10.0.0.61:8082/domains
2025-10-31 00:36:02,489 - DEBUG - Requesting: http://10.0.0.61:8082/domain/baeb4ece-e0da-438d-906b-54b1de16acc9
2025-10-31 00:36:02,496 - DEBUG - Requesting: http://10.0.0.61:8082/project/e1aa05eb-4bfb-4f92-ac9d-9c7f2e3c2aa3
2025-10-31 00:36:02,545 - INFO - Validated that VN __public__ is not found in API Server
2025-10-31 00:36:02,545 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-10-31 00:36:02,551 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-10-31 00:36:02,560 - DEBUG - VN __public__ is not present in Agent 10.0.0.40
2025-10-31 00:36:02,560 - INFO - Validated that VN __public__ is not in any agent
2025-10-31 00:36:02,560 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-10-31 00:36:02,569 - DEBUG - Requesting: http://10.0.0.61:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-10-31 00:36:02,577 - DEBUG - Requesting: http://10.0.0.61:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-10-31 00:36:02,582 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-10-31 00:36:02,583 - INFO - Deleting pod default:ctest-nginx-pod-94011549
2025-10-31 00:36:02,613 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=dd882b0e-4924-4009-9759-3c914411dac8
2025-10-31 00:36:02,625 - WARNING - Pod uuid dd882b0e-4924-4009-9759-3c914411dac8 is still seen in agent 10.0.0.40 VM list
2025-10-31 00:36:07,627 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=dd882b0e-4924-4009-9759-3c914411dac8
2025-10-31 00:36:07,633 - DEBUG - Pod dd882b0e-4924-4009-9759-3c914411dac8 is not in agent 10.0.0.40 VM list
2025-10-31 00:36:07,633 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:72b82dea-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:36:07,638 - DEBUG - VMI 72b82dea-b5f1-11f0-8589-fa163efc4e20 is removed from agent 10.0.0.40
2025-10-31 00:36:07,638 - INFO - Verified that pod ctest-nginx-pod-94011549 is removed in agent
2025-10-31 00:36:07,639 - INFO - Deleting pod default:ctest-nginx-pod-18649308
2025-10-31 00:36:07,657 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=83ef1ec7-cb81-4eba-bf9f-d182180dd961
2025-10-31 00:36:07,671 - WARNING - Pod uuid 83ef1ec7-cb81-4eba-bf9f-d182180dd961 is still seen in agent 10.0.0.40 VM list
2025-10-31 00:36:12,671 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=83ef1ec7-cb81-4eba-bf9f-d182180dd961
2025-10-31 00:36:12,683 - DEBUG - Pod 83ef1ec7-cb81-4eba-bf9f-d182180dd961 is not in agent 10.0.0.40 VM list
2025-10-31 00:36:12,683 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:71a53434-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:36:12,688 - DEBUG - VMI 71a53434-b5f1-11f0-8589-fa163efc4e20 is removed from agent 10.0.0.40
2025-10-31 00:36:12,688 - INFO - Verified that pod ctest-nginx-pod-18649308 is removed in agent
2025-10-31 00:36:12,688 - INFO - Deleting service : ctest-nginx-svc-40951002
2025-10-31 00:36:12,982 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AgentXmppConnectionStatusReq?
2025-10-31 00:36:12,989 - DEBUG - No XMPP flaps were noticed during the test
2025-10-31 00:36:12,989 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:10]
2025-10-31 00:36:12,989 - INFO - --------------------------------------------------------------------------------
2025-10-31 00:36:19,459 - DEBUG - [10.0.0.61]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-10-31 00:36:19,586 - DEBUG - Output : cluster_project={}
2025-10-31 00:36:19,640 - DEBUG - [10.0.0.61]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-10-31 00:36:19,812 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-10-31 00:36:19,817 - DEBUG - [10.0.0.61]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-10-31 00:36:19,946 - DEBUG - Output : cluster_project={}
2025-10-31 00:36:35,055 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:36:35,143 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:36:35,183 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:36:35,273 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:36:35,351 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:36:35,402 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:36:35,486 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:36:40,532 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-10-31 00:36:40,772 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AgentXmppConnectionStatusReq?
2025-10-31 00:36:43,601 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_ApplicationPolicySetReq?
2025-10-31 00:36:43,618 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AclReq?
2025-10-31 00:36:43,657 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AclReq?
2025-10-31 00:36:45,685 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AclReq?
2025-10-31 00:39:45,879 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AgentXmppConnectionStatusReq?
2025-10-31 00:39:45,937 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=e04f0f75-85ee-435c-8b3e-617ba3074ded
2025-10-31 00:39:45,945 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=86bdcaf5-3111-495c-8960-643a39c95eeb
2025-10-31 00:39:45,972 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=0cc93ff9-7787-4414-a055-f63e4d237e52
2025-10-31 00:39:45,983 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=b2ffa322-5a62-437d-90cc-c27f61265f0e
2025-10-31 00:39:45,988 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=19236f99-159f-4888-9dfa-c7f74803e904
2025-10-31 00:39:45,994 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=45b69348-8bc6-494d-b9b1-3ba75395005a
2025-10-31 00:39:46,001 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=16fcbb06-2cbd-45a9-b7ca-2e5be6249c44
2025-10-31 00:39:46,007 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=79cd9cc5-a3b7-4929-ba5f-bc5b58d0a9ff
2025-10-31 00:39:51,013 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=86bdcaf5-3111-495c-8960-643a39c95eeb
2025-10-31 00:39:51,015 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=e04f0f75-85ee-435c-8b3e-617ba3074ded
2025-10-31 00:39:51,022 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a09a9e00-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:39:51,025 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9f0fee78-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:39:51,036 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=19236f99-159f-4888-9dfa-c7f74803e904
2025-10-31 00:39:51,038 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=b2ffa322-5a62-437d-90cc-c27f61265f0e
2025-10-31 00:39:51,039 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=0cc93ff9-7787-4414-a055-f63e4d237e52
2025-10-31 00:39:51,041 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=79cd9cc5-a3b7-4929-ba5f-bc5b58d0a9ff
2025-10-31 00:39:51,042 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=16fcbb06-2cbd-45a9-b7ca-2e5be6249c44
2025-10-31 00:39:51,045 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=45b69348-8bc6-494d-b9b1-3ba75395005a
2025-10-31 00:39:51,060 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a01e1402-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:39:51,061 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9f8302fa-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:39:56,060 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=0cc93ff9-7787-4414-a055-f63e4d237e52
2025-10-31 00:39:56,063 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=b2ffa322-5a62-437d-90cc-c27f61265f0e
2025-10-31 00:39:56,065 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=45b69348-8bc6-494d-b9b1-3ba75395005a
2025-10-31 00:39:56,066 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=79cd9cc5-a3b7-4929-ba5f-bc5b58d0a9ff
2025-10-31 00:40:01,070 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=0cc93ff9-7787-4414-a055-f63e4d237e52
2025-10-31 00:40:01,074 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=45b69348-8bc6-494d-b9b1-3ba75395005a
2025-10-31 00:40:01,075 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=b2ffa322-5a62-437d-90cc-c27f61265f0e
2025-10-31 00:40:01,076 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=79cd9cc5-a3b7-4929-ba5f-bc5b58d0a9ff
2025-10-31 00:40:06,079 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=0cc93ff9-7787-4414-a055-f63e4d237e52
2025-10-31 00:40:06,086 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=45b69348-8bc6-494d-b9b1-3ba75395005a
2025-10-31 00:40:06,088 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=79cd9cc5-a3b7-4929-ba5f-bc5b58d0a9ff
2025-10-31 00:40:06,089 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=b2ffa322-5a62-437d-90cc-c27f61265f0e
2025-10-31 00:40:11,090 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=0cc93ff9-7787-4414-a055-f63e4d237e52
2025-10-31 00:40:11,099 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=45b69348-8bc6-494d-b9b1-3ba75395005a
2025-10-31 00:40:11,101 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=b2ffa322-5a62-437d-90cc-c27f61265f0e
2025-10-31 00:40:11,102 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=79cd9cc5-a3b7-4929-ba5f-bc5b58d0a9ff
2025-10-31 00:40:16,098 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=0cc93ff9-7787-4414-a055-f63e4d237e52
2025-10-31 00:40:16,111 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=79cd9cc5-a3b7-4929-ba5f-bc5b58d0a9ff
2025-10-31 00:40:16,112 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=45b69348-8bc6-494d-b9b1-3ba75395005a
2025-10-31 00:40:16,114 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=b2ffa322-5a62-437d-90cc-c27f61265f0e
2025-10-31 00:40:21,118 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=0cc93ff9-7787-4414-a055-f63e4d237e52
2025-10-31 00:40:21,125 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a11fd9f8-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:40:21,126 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=79cd9cc5-a3b7-4929-ba5f-bc5b58d0a9ff
2025-10-31 00:40:21,128 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=45b69348-8bc6-494d-b9b1-3ba75395005a
2025-10-31 00:40:21,128 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=b2ffa322-5a62-437d-90cc-c27f61265f0e
2025-10-31 00:40:21,140 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a1f4d446-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:40:21,143 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a188f636-b5f1-11f0-8589-fa163efc4e20
2025-10-31 00:40:21,144 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a25809e4-b5f1-11f0-8589-fa163efc4e20