2025-01-14 06:30:49,485 - DEBUG - [10.0.0.22]: Running cmd : hostname
2025-01-14 06:30:50,203 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-923-1
2025-01-14 06:30:50,204 - DEBUG - [10.0.0.22]: Running cmd : hostname -f
2025-01-14 06:30:50,257 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-923-1
2025-01-14 06:30:50,257 - DEBUG - [10.0.0.22]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-01-14 06:30:50,405 - DEBUG - Output : NAMES
contrail_test_vzc6IJFEg
contrail-kubernetes-master_kubemanager_1
analytics_snmp_topology_1
analytics_snmp_snmp-collector_1
analytics_snmp_nodemgr_1
analytics_alarm_kafka_1
analytics_alarm_nodemgr_1
analytics_alarm_alarm-gen_1
analytics_nodemgr_1
analytics_collector_1
analytics_api_1
analytics_database_query-engine_1
analytics_database_cassandra_1
analytics_database_nodemgr_1
redis_redis_1
webui_job_1
webui_web_1
control_named_1
control_control_1
control_nodemgr_1
control_dns_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_cassandra_1
config_database_rabbitmq_1
config_api_devicemgr_1
config_api_api_1
config_api_svcmonitor_1
config_api_schema_1
config_api_nodemgr_1
config_api_dnsmasq_1
2025-01-14 06:30:50,405 - DEBUG - [10.0.0.22]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-01-14 06:30:50,450 - DEBUG - Output : 10.0.0.22/24
172.17.0.1/16
2025-01-14 06:30:50,450 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}'
2025-01-14 06:30:50,494 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-923-1
2025-01-14 06:30:50,603 - DEBUG - [10.0.0.50]: Running cmd : hostname
2025-01-14 06:30:50,844 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-923-1
2025-01-14 06:30:50,844 - DEBUG - [10.0.0.50]: Running cmd : hostname -f
2025-01-14 06:30:50,880 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-923-1
2025-01-14 06:30:50,880 - DEBUG - [10.0.0.50]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-01-14 06:30:50,925 - DEBUG - Output : NAMES
vrouter_nodemgr_1
vrouter_vrouter-agent_1
2025-01-14 06:30:50,925 - DEBUG - [10.0.0.50]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-01-14 06:30:51,005 - DEBUG - Output : 10.0.0.50/24
2025-01-14 06:30:51,006 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.0.0.50 | head -n 1 | awk '{print $2}'
2025-01-14 06:30:51,040 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-923-1
2025-01-14 06:30:51,149 - INFO - Using existing project ['default-domain', 'k8s-default'](554ead42-7e36-4831-9f6b-c6ddc94c8e34)
2025-01-14 06:30:51,282 - INFO - Using existing project ['default-domain', 'k8s-default'](554ead42-7e36-4831-9f6b-c6ddc94c8e34)
2025-01-14 06:30:51,447 - INFO - ================================================================================
2025-01-14 06:30:51,448 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-01-14 06:30:51,449 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-01-14 06:30:51,815 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2025-01-14 06:30:51,825 - DEBUG - Nothing to compare xmpp stats {'10.0.0.50': {'10.0.0.22': '0'}} with
2025-01-14 06:30:51,825 - INFO - Initial checks done. Running the testcase now
2025-01-14 06:30:51,826 - INFO -
2025-01-14 06:30:51,833 - INFO - Namespace default already exists
2025-01-14 06:30:51,834 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-01-14 06:30:52,248 - DEBUG - Output : cluster_project={}
2025-01-14 06:30:52,357 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-01-14 06:30:52,357 - DEBUG - Project uuid in Contrail is 554ead42-7e36-4831-9f6b-c6ddc94c8e34
2025-01-14 06:30:52,358 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-01-14 06:30:52,498 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-01-14 06:30:52,498 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_NamespaceDatabaseList?namespace_uuid=df1cff84-e356-411d-aa87-b0e8e5fb3f2e
2025-01-14 06:30:52,503 - INFO - Namespace default with uuid df1cff84-e356-411d-aa87-b0e8e5fb3f2e found in kube manager
2025-01-14 06:30:52,503 - INFO - Namespace default verification passed
2025-01-14 06:30:52,507 - DEBUG - Service ctest-nginx-svc-70573064 not present
2025-01-14 06:30:52,507 - INFO - Creating service ctest-nginx-svc-70573064
2025-01-14 06:30:52,533 - INFO - Created Service ctest-nginx-svc-70573064
2025-01-14 06:30:52,539 - DEBUG - Pod ctest-nginx-pod-50608819 not present
2025-01-14 06:30:52,540 - INFO - Creating Pod ctest-nginx-pod-50608819
2025-01-14 06:30:52,590 - DEBUG - Node for Pod ctest-nginx-pod-50608819 not yet populated
2025-01-14 06:30:55,598 - DEBUG - Pod : ctest-nginx-pod-50608819 UUID is 4ab90162-ed93-4e89-8cd2-d52ba9426885
2025-01-14 06:30:55,602 - DEBUG - Pod ctest-nginx-pod-46487101 not present
2025-01-14 06:30:55,602 - INFO - Creating Pod ctest-nginx-pod-46487101
2025-01-14 06:30:55,647 - DEBUG - Pod : ctest-nginx-pod-46487101 UUID is 910a011d-ecf8-4310-a7c0-78330bb05035
2025-01-14 06:30:55,937 - INFO - Created VN __public__, UUID :ffdc9f56-2de1-4db2-a41c-eefdfc15f274
2025-01-14 06:30:56,298 - DEBUG - Requesting: http://10.0.0.22:8082/domains
2025-01-14 06:30:56,305 - DEBUG - Requesting: http://10.0.0.22:8082/domain/6aaa5b6c-43c5-45bd-89ef-8eab96531502
2025-01-14 06:30:56,327 - DEBUG - Requesting: http://10.0.0.22:8082/project/554ead42-7e36-4831-9f6b-c6ddc94c8e34
2025-01-14 06:30:56,482 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/ffdc9f56-2de1-4db2-a41c-eefdfc15f274
2025-01-14 06:30:56,496 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/ffdc9f56-2de1-4db2-a41c-eefdfc15f274
2025-01-14 06:30:56,510 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/6e85ccdf-bf29-4ebe-be67-1520278d73c0
2025-01-14 06:30:56,518 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/6e85ccdf-bf29-4ebe-be67-1520278d73c0
2025-01-14 06:30:56,528 - DEBUG - Requesting: http://10.0.0.22:8082/route-target/8cfc1e2f-a818-4c9d-9730-6ef174a59a85
2025-01-14 06:30:56,549 - DEBUG - Route Targets: ['target:64512:8000006']
2025-01-14 06:30:56,549 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/ffdc9f56-2de1-4db2-a41c-eefdfc15f274
2025-01-14 06:30:56,565 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/6e85ccdf-bf29-4ebe-be67-1520278d73c0
2025-01-14 06:30:56,649 - INFO - Verified VN network id 13 for VN ffdc9f56-2de1-4db2-a41c-eefdfc15f274
2025-01-14 06:30:56,649 - INFO - Verifications in API Server for VN __public__ passed
2025-01-14 06:30:56,649 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/ffdc9f56-2de1-4db2-a41c-eefdfc15f274
2025-01-14 06:30:56,663 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/6e85ccdf-bf29-4ebe-be67-1520278d73c0
2025-01-14 06:30:56,676 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/6e85ccdf-bf29-4ebe-be67-1520278d73c0
2025-01-14 06:30:56,692 - DEBUG - Requesting: http://10.0.0.22:8082/route-target/8cfc1e2f-a818-4c9d-9730-6ef174a59a85
2025-01-14 06:30:56,705 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-01-14 06:30:56,714 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '18436786167033712050', 'uuid-lslong': '11825589496191513204'}, 'enable': 'true', 'created': '2025-01-14T06:30:55', 'last-modified': '2025-01-14T06:30:56', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.543822'}
2025-01-14 06:30:56,714 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-01-14 06:30:56,728 - DEBUG - Route Targets: ['target:64512:8000006']
2025-01-14 06:30:56,728 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-01-14 06:30:56,729 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-01-14 06:30:56,729 - DEBUG - Requesting: http://10.0.0.22:8082/domains
2025-01-14 06:30:56,738 - DEBUG - Requesting: http://10.0.0.22:8082/domain/6aaa5b6c-43c5-45bd-89ef-8eab96531502
2025-01-14 06:30:56,747 - DEBUG - Requesting: http://10.0.0.22:8082/project/554ead42-7e36-4831-9f6b-c6ddc94c8e34
2025-01-14 06:30:56,803 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/ffdc9f56-2de1-4db2-a41c-eefdfc15f274
2025-01-14 06:30:56,816 - DEBUG - =>VN __public__ has no policy to be verified
2025-01-14 06:30:56,816 - DEBUG - Verifying the vn in opserver
2025-01-14 06:30:56,816 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.22
2025-01-14 06:30:56,816 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-01-14 06:30:56,832 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-01-14 06:30:56,832 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-01-14 06:30:56,832 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-01-14 06:30:56,842 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-01-14 06:30:56,860 - DEBUG - Do not have enough data to verify VN in agent
2025-01-14 06:30:56,860 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-01-14 06:30:56,870 - DEBUG - VRF ids for VN __public__: {}
2025-01-14 06:30:56,992 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-01-14 06:30:57,113 - DEBUG - Requesting: http://10.0.0.22:8082/domains
2025-01-14 06:30:57,121 - DEBUG - Requesting: http://10.0.0.22:8082/domain/6aaa5b6c-43c5-45bd-89ef-8eab96531502
2025-01-14 06:30:57,131 - DEBUG - Requesting: http://10.0.0.22:8082/project/554ead42-7e36-4831-9f6b-c6ddc94c8e34
2025-01-14 06:30:57,184 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/ffdc9f56-2de1-4db2-a41c-eefdfc15f274
2025-01-14 06:30:57,198 - DEBUG - Requesting: http://10.0.0.22:8082/floating-ip-pool/992b7a68-21b0-4998-9bb0-5866499da5d3
2025-01-14 06:30:57,206 - DEBUG - Requesting: http://10.0.0.22:8082/domains
2025-01-14 06:30:57,217 - DEBUG - Requesting: http://10.0.0.22:8082/domain/6aaa5b6c-43c5-45bd-89ef-8eab96531502
2025-01-14 06:30:57,229 - DEBUG - Requesting: http://10.0.0.22:8082/project/554ead42-7e36-4831-9f6b-c6ddc94c8e34
2025-01-14 06:30:57,277 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/ffdc9f56-2de1-4db2-a41c-eefdfc15f274
2025-01-14 06:30:57,290 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-01-14 06:30:57,290 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-01-14 06:30:57,299 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-01-14 06:30:57,307 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-01-14 06:30:57,307 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-01-14 06:30:57,312 - DEBUG - Ingress ctest-nginx-ingress-65643232 not present
2025-01-14 06:30:57,313 - INFO - Creating Ingress ctest-nginx-ingress-65643232
2025-01-14 06:30:57,322 - INFO - Created Ingress ctest-nginx-ingress-65643232
2025-01-14 06:30:57,327 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-65643232
2025-01-14 06:31:00,334 - DEBUG - For Ingress ctest-nginx-ingress-65643232, Cluster IP: 10.47.255.249, External IPs ['24.153.142.188']
2025-01-14 06:31:00,334 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-65643232
2025-01-14 06:31:00,335 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=1056a6aa-af92-466c-bd16-310aed0061c8
2025-01-14 06:31:00,347 - INFO - Ingress ctest-nginx-ingress-65643232 with uuid 1056a6aa-af92-466c-bd16-310aed0061c8 found in kube manager
2025-01-14 06:31:00,348 - INFO - Ingress ctest-nginx-ingress-65643232 verification passed
2025-01-14 06:31:00,351 - DEBUG - Pod ctest-busybox-pod-52693135 not present
2025-01-14 06:31:00,351 - INFO - Creating Pod ctest-busybox-pod-52693135
2025-01-14 06:31:00,406 - DEBUG - Pod : ctest-busybox-pod-52693135 UUID is c535796f-72cd-4532-8404-3888e38f6316
2025-01-14 06:31:00,413 - DEBUG - Pod ctest-nginx-pod-50608819 not in running state.Currently in Pending
2025-01-14 06:31:05,422 - DEBUG - Pod ctest-nginx-pod-50608819 not in running state.Currently in Pending
2025-01-14 06:31:10,440 - INFO - Pod ctest-nginx-pod-50608819 is in running state.Got IP 10.47.255.251
2025-01-14 06:31:10,501 - DEBUG - Pod ctest-nginx-pod-50608819 has vmi 1b4d4048-d241-11ef-adff-fa163e428b74
2025-01-14 06:31:10,502 - INFO - Verified pod ctest-nginx-pod-50608819 in contrail-api
2025-01-14 06:31:10,502 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:31:10,517 - DEBUG - VMI 1b4d4048-d241-11ef-adff-fa163e428b74 is active in agent 10.0.0.50
2025-01-14 06:31:10,517 - INFO - Verified Pod ctest-nginx-pod-50608819 in agent 10.0.0.50
2025-01-14 06:31:10,517 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=4ab90162-ed93-4e89-8cd2-d52ba9426885
2025-01-14 06:31:10,523 - INFO - Pod ctest-nginx-pod-50608819 with uuid 4ab90162-ed93-4e89-8cd2-d52ba9426885 found in kube manager
2025-01-14 06:31:10,523 - INFO - Pod ctest-nginx-pod-50608819 verification passed
2025-01-14 06:31:10,604 - DEBUG - [Pod ctest-nginx-pod-50608819] Cmd: echo ctest-nginx-pod-50608819 > /usr/share/nginx/html/index.html, Output:
2025-01-14 06:31:10,611 - INFO - Pod ctest-nginx-pod-46487101 is in running state.Got IP 10.47.255.250
2025-01-14 06:31:10,637 - DEBUG - Pod ctest-nginx-pod-46487101 has vmi 1c236e98-d241-11ef-adff-fa163e428b74
2025-01-14 06:31:10,637 - INFO - Verified pod ctest-nginx-pod-46487101 in contrail-api
2025-01-14 06:31:10,637 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:31:10,649 - DEBUG - VMI 1c236e98-d241-11ef-adff-fa163e428b74 is active in agent 10.0.0.50
2025-01-14 06:31:10,650 - INFO - Verified Pod ctest-nginx-pod-46487101 in agent 10.0.0.50
2025-01-14 06:31:10,650 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=910a011d-ecf8-4310-a7c0-78330bb05035
2025-01-14 06:31:10,656 - INFO - Pod ctest-nginx-pod-46487101 with uuid 910a011d-ecf8-4310-a7c0-78330bb05035 found in kube manager
2025-01-14 06:31:10,656 - INFO - Pod ctest-nginx-pod-46487101 verification passed
2025-01-14 06:31:10,723 - DEBUG - [Pod ctest-nginx-pod-46487101] Cmd: echo ctest-nginx-pod-46487101 > /usr/share/nginx/html/index.html, Output:
2025-01-14 06:31:10,729 - INFO - Pod ctest-busybox-pod-52693135 is in running state.Got IP 10.47.255.248
2025-01-14 06:31:10,747 - DEBUG - Pod ctest-busybox-pod-52693135 has vmi 1f8c1ca6-d241-11ef-adff-fa163e428b74
2025-01-14 06:31:10,747 - INFO - Verified pod ctest-busybox-pod-52693135 in contrail-api
2025-01-14 06:31:10,748 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:31:10,761 - DEBUG - VMI 1f8c1ca6-d241-11ef-adff-fa163e428b74 is active in agent 10.0.0.50
2025-01-14 06:31:10,761 - INFO - Verified Pod ctest-busybox-pod-52693135 in agent 10.0.0.50
2025-01-14 06:31:10,761 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=c535796f-72cd-4532-8404-3888e38f6316
2025-01-14 06:31:10,766 - INFO - Pod ctest-busybox-pod-52693135 with uuid c535796f-72cd-4532-8404-3888e38f6316 found in kube manager
2025-01-14 06:31:10,767 - INFO - Pod ctest-busybox-pod-52693135 verification passed
2025-01-14 06:31:10,830 - DEBUG - [Pod ctest-busybox-pod-52693135] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-46487101
2025-01-14 06:31:10,830 - DEBUG - [Pod Pod ctest-busybox-pod-52693135] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-01-14 06:31:10,830 - DEBUG - [Pod Pod ctest-busybox-pod-52693135] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-46487101
2025-01-14 06:31:10,884 - DEBUG - [Pod ctest-busybox-pod-52693135] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-50608819
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-01-14 06:31:10,884 - DEBUG - [Pod Pod ctest-busybox-pod-52693135] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-01-14 06:31:10,885 - DEBUG - [Pod Pod ctest-busybox-pod-52693135] Cmd output: ctest-nginx-pod-50608819
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-01-14 06:31:10,885 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-50608819': 1, 'ctest-nginx-pod-46487101': 1}
2025-01-14 06:31:10,885 - WARNING - Nothing to delete parallely
2025-01-14 06:31:10,885 - INFO - Deleting pod default:ctest-busybox-pod-52693135
2025-01-14 06:31:10,904 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=c535796f-72cd-4532-8404-3888e38f6316
2025-01-14 06:31:10,914 - WARNING - Pod uuid c535796f-72cd-4532-8404-3888e38f6316 is still seen in agent 10.0.0.50 VM list
2025-01-14 06:31:15,916 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=c535796f-72cd-4532-8404-3888e38f6316
2025-01-14 06:31:15,926 - WARNING - Pod uuid c535796f-72cd-4532-8404-3888e38f6316 is still seen in agent 10.0.0.50 VM list
2025-01-14 06:31:20,927 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=c535796f-72cd-4532-8404-3888e38f6316
2025-01-14 06:31:20,938 - WARNING - Pod uuid c535796f-72cd-4532-8404-3888e38f6316 is still seen in agent 10.0.0.50 VM list
2025-01-14 06:31:25,939 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=c535796f-72cd-4532-8404-3888e38f6316
2025-01-14 06:31:25,947 - WARNING - Pod uuid c535796f-72cd-4532-8404-3888e38f6316 is still seen in agent 10.0.0.50 VM list
2025-01-14 06:31:30,948 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=c535796f-72cd-4532-8404-3888e38f6316
2025-01-14 06:31:30,955 - WARNING - Pod uuid c535796f-72cd-4532-8404-3888e38f6316 is still seen in agent 10.0.0.50 VM list
2025-01-14 06:31:35,956 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=c535796f-72cd-4532-8404-3888e38f6316
2025-01-14 06:31:35,983 - WARNING - Pod uuid c535796f-72cd-4532-8404-3888e38f6316 is still seen in agent 10.0.0.50 VM list
2025-01-14 06:31:40,984 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=c535796f-72cd-4532-8404-3888e38f6316
2025-01-14 06:31:40,995 - WARNING - Pod uuid c535796f-72cd-4532-8404-3888e38f6316 is still seen in agent 10.0.0.50 VM list
2025-01-14 06:31:45,996 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=c535796f-72cd-4532-8404-3888e38f6316
2025-01-14 06:31:46,005 - DEBUG - Pod c535796f-72cd-4532-8404-3888e38f6316 is not in agent 10.0.0.50 VM list
2025-01-14 06:31:46,005 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1f8c1ca6-d241-11ef-adff-fa163e428b74
2025-01-14 06:31:46,013 - DEBUG - VMI 1f8c1ca6-d241-11ef-adff-fa163e428b74 is removed from agent 10.0.0.50
2025-01-14 06:31:46,013 - INFO - Verified that pod ctest-busybox-pod-52693135 is removed in agent
2025-01-14 06:31:46,013 - INFO - Deleting Ingress : ctest-nginx-ingress-65643232
2025-01-14 06:31:46,021 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=1056a6aa-af92-466c-bd16-310aed0061c8
2025-01-14 06:31:46,033 - ERROR - Ingress ctest-nginx-ingress-65643232 with uuid 1056a6aa-af92-466c-bd16-310aed0061c8 still found in kube manager
2025-01-14 06:31:47,036 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=1056a6aa-af92-466c-bd16-310aed0061c8
2025-01-14 06:31:47,043 - INFO - Ingress ctest-nginx-ingress-65643232 with uuid 1056a6aa-af92-466c-bd16-310aed0061c8 deleted successfully from kube manager
2025-01-14 06:31:47,044 - INFO - Verifications on Ingress %s cleanup passed
2025-01-14 06:31:47,044 - INFO - Deleting the FIP pool __fip_pool_public__
2025-01-14 06:31:47,262 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-01-14 06:31:47,273 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-01-14 06:31:47,273 - INFO - Deleting VN __public__
2025-01-14 06:31:47,725 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/6e85ccdf-bf29-4ebe-be67-1520278d73c0
2025-01-14 06:31:47,738 - DEBUG - Response Code: 404
2025-01-14 06:31:47,738 - DEBUG - Requesting: http://10.0.0.22:8082/domains
2025-01-14 06:31:47,749 - DEBUG - Requesting: http://10.0.0.22:8082/domain/6aaa5b6c-43c5-45bd-89ef-8eab96531502
2025-01-14 06:31:47,761 - DEBUG - Requesting: http://10.0.0.22:8082/project/554ead42-7e36-4831-9f6b-c6ddc94c8e34
2025-01-14 06:31:47,848 - INFO - Validated that VN __public__ is not found in API Server
2025-01-14 06:31:47,848 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-01-14 06:31:47,865 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-01-14 06:31:47,878 - DEBUG - VN __public__ is not present in Agent 10.0.0.50
2025-01-14 06:31:47,878 - INFO - Validated that VN __public__ is not in any agent
2025-01-14 06:31:47,878 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-01-14 06:31:47,891 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-01-14 06:31:47,914 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-01-14 06:31:47,954 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-01-14 06:31:47,955 - INFO - Deleting pod default:ctest-nginx-pod-46487101
2025-01-14 06:31:47,985 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=910a011d-ecf8-4310-a7c0-78330bb05035
2025-01-14 06:31:48,014 - WARNING - Pod uuid 910a011d-ecf8-4310-a7c0-78330bb05035 is still seen in agent 10.0.0.50 VM list
2025-01-14 06:31:53,015 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=910a011d-ecf8-4310-a7c0-78330bb05035
2025-01-14 06:31:53,025 - DEBUG - Pod 910a011d-ecf8-4310-a7c0-78330bb05035 is not in agent 10.0.0.50 VM list
2025-01-14 06:31:53,026 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1c236e98-d241-11ef-adff-fa163e428b74
2025-01-14 06:31:53,034 - DEBUG - VMI 1c236e98-d241-11ef-adff-fa163e428b74 is removed from agent 10.0.0.50
2025-01-14 06:31:53,034 - INFO - Verified that pod ctest-nginx-pod-46487101 is removed in agent
2025-01-14 06:31:53,034 - INFO - Deleting pod default:ctest-nginx-pod-50608819
2025-01-14 06:31:53,050 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=4ab90162-ed93-4e89-8cd2-d52ba9426885
2025-01-14 06:31:53,058 - WARNING - Pod uuid 4ab90162-ed93-4e89-8cd2-d52ba9426885 is still seen in agent 10.0.0.50 VM list
2025-01-14 06:31:58,059 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=4ab90162-ed93-4e89-8cd2-d52ba9426885
2025-01-14 06:31:58,067 - DEBUG - Pod 4ab90162-ed93-4e89-8cd2-d52ba9426885 is not in agent 10.0.0.50 VM list
2025-01-14 06:31:58,067 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1b4d4048-d241-11ef-adff-fa163e428b74
2025-01-14 06:31:58,075 - DEBUG - VMI 1b4d4048-d241-11ef-adff-fa163e428b74 is removed from agent 10.0.0.50
2025-01-14 06:31:58,076 - INFO - Verified that pod ctest-nginx-pod-50608819 is removed in agent
2025-01-14 06:31:58,076 - INFO - Deleting service : ctest-nginx-svc-70573064
2025-01-14 06:31:58,370 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2025-01-14 06:31:58,381 - DEBUG - No XMPP flaps were noticed during the test
2025-01-14 06:31:58,381 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:07]
2025-01-14 06:31:58,381 - INFO - --------------------------------------------------------------------------------
2025-01-14 06:32:05,020 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-01-14 06:32:05,164 - DEBUG - Output : cluster_project={}
2025-01-14 06:32:05,279 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-01-14 06:32:05,483 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-01-14 06:32:05,497 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-01-14 06:32:05,675 - DEBUG - Output : cluster_project={}
2025-01-14 06:32:20,845 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:32:20,893 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:32:20,983 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:32:21,078 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:32:26,130 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:32:26,221 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:32:26,270 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:32:26,362 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-01-14 06:32:26,643 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2025-01-14 06:32:29,705 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_ApplicationPolicySetReq?
2025-01-14 06:32:29,720 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AclReq?
2025-01-14 06:32:29,753 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AclReq?
2025-01-14 06:32:31,788 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AclReq?
2025-01-14 06:35:38,456 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2025-01-14 06:35:38,497 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1b4bfb70-e735-4b94-891f-fbd948beaa49
2025-01-14 06:35:38,548 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=7bf13a36-8fe8-4519-a0a2-0dc377426280
2025-01-14 06:35:38,565 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1805443e-cccf-49a7-9d3c-44d3ca207ee4
2025-01-14 06:35:38,576 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8e04cc6b-5ab7-4ab1-8f7b-b22bd243cac9
2025-01-14 06:35:38,585 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f5befafd-25ce-45c1-a61c-8a8189c93400
2025-01-14 06:35:38,590 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=6c61bcba-85c4-4569-b149-0abd1042e572
2025-01-14 06:35:38,595 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=a24b17ed-d718-433e-9f3a-c81e00f51383
2025-01-14 06:35:38,601 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=07363803-e577-4ba5-97f6-dacedf492329
2025-01-14 06:35:43,614 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=7bf13a36-8fe8-4519-a0a2-0dc377426280
2025-01-14 06:35:43,616 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1805443e-cccf-49a7-9d3c-44d3ca207ee4
2025-01-14 06:35:43,617 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1b4bfb70-e735-4b94-891f-fbd948beaa49
2025-01-14 06:35:43,629 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:46fa3976-d241-11ef-adff-fa163e428b74
2025-01-14 06:35:43,633 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:467c03f8-d241-11ef-adff-fa163e428b74
2025-01-14 06:35:43,647 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8e04cc6b-5ab7-4ab1-8f7b-b22bd243cac9
2025-01-14 06:35:43,649 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=a24b17ed-d718-433e-9f3a-c81e00f51383
2025-01-14 06:35:43,659 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:47bf3b86-d241-11ef-adff-fa163e428b74
2025-01-14 06:35:43,665 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=6c61bcba-85c4-4569-b149-0abd1042e572
2025-01-14 06:35:43,669 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=07363803-e577-4ba5-97f6-dacedf492329
2025-01-14 06:35:43,670 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f5befafd-25ce-45c1-a61c-8a8189c93400
2025-01-14 06:35:43,687 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:484a590a-d241-11ef-adff-fa163e428b74
2025-01-14 06:35:48,633 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1805443e-cccf-49a7-9d3c-44d3ca207ee4
2025-01-14 06:35:48,665 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=a24b17ed-d718-433e-9f3a-c81e00f51383
2025-01-14 06:35:48,688 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=07363803-e577-4ba5-97f6-dacedf492329
2025-01-14 06:35:48,690 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f5befafd-25ce-45c1-a61c-8a8189c93400
2025-01-14 06:35:53,642 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1805443e-cccf-49a7-9d3c-44d3ca207ee4
2025-01-14 06:35:53,679 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=a24b17ed-d718-433e-9f3a-c81e00f51383
2025-01-14 06:35:53,701 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=07363803-e577-4ba5-97f6-dacedf492329
2025-01-14 06:35:53,704 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f5befafd-25ce-45c1-a61c-8a8189c93400
2025-01-14 06:35:58,653 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1805443e-cccf-49a7-9d3c-44d3ca207ee4
2025-01-14 06:35:58,687 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=a24b17ed-d718-433e-9f3a-c81e00f51383
2025-01-14 06:35:58,716 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=07363803-e577-4ba5-97f6-dacedf492329
2025-01-14 06:35:58,719 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f5befafd-25ce-45c1-a61c-8a8189c93400
2025-01-14 06:36:03,662 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1805443e-cccf-49a7-9d3c-44d3ca207ee4
2025-01-14 06:36:03,699 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=a24b17ed-d718-433e-9f3a-c81e00f51383
2025-01-14 06:36:03,730 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=07363803-e577-4ba5-97f6-dacedf492329
2025-01-14 06:36:03,732 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f5befafd-25ce-45c1-a61c-8a8189c93400
2025-01-14 06:36:08,670 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1805443e-cccf-49a7-9d3c-44d3ca207ee4
2025-01-14 06:36:08,707 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=a24b17ed-d718-433e-9f3a-c81e00f51383
2025-01-14 06:36:08,745 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f5befafd-25ce-45c1-a61c-8a8189c93400
2025-01-14 06:36:08,747 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=07363803-e577-4ba5-97f6-dacedf492329
2025-01-14 06:36:13,680 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1805443e-cccf-49a7-9d3c-44d3ca207ee4
2025-01-14 06:36:13,688 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:49d84868-d241-11ef-adff-fa163e428b74
2025-01-14 06:36:13,717 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=a24b17ed-d718-433e-9f3a-c81e00f51383
2025-01-14 06:36:13,725 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4a6e59a2-d241-11ef-adff-fa163e428b74
2025-01-14 06:36:13,758 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f5befafd-25ce-45c1-a61c-8a8189c93400
2025-01-14 06:36:13,760 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=07363803-e577-4ba5-97f6-dacedf492329
2025-01-14 06:36:13,771 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:48f95e3c-d241-11ef-adff-fa163e428b74
2025-01-14 06:36:13,774 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4b280852-d241-11ef-adff-fa163e428b74