2024-11-04 06:40:11,386 - DEBUG - [10.0.0.24]: Running cmd : hostname
2024-11-04 06:40:11,621 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-759-1
2024-11-04 06:40:11,621 - DEBUG - [10.0.0.24]: Running cmd : hostname -f
2024-11-04 06:40:11,655 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-759-1
2024-11-04 06:40:11,655 - DEBUG - [10.0.0.24]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2024-11-04 06:40:11,752 - DEBUG - Output : NAMES
contrail_test_E2wGl2gNl
contrail-kubernetes-master_kubemanager_1
analytics_database_cassandra_1
analytics_database_nodemgr_1
analytics_database_query-engine_1
analytics_snmp_topology_1
analytics_snmp_snmp-collector_1
analytics_snmp_nodemgr_1
analytics_alarm_kafka_1
analytics_alarm_nodemgr_1
analytics_alarm_alarm-gen_1
analytics_nodemgr_1
analytics_api_1
analytics_collector_1
redis_redis_1
webui_job_1
webui_web_1
control_named_1
control_control_1
control_nodemgr_1
control_dns_1
config_database_cassandra_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_rabbitmq_1
config_api_devicemgr_1
config_api_api_1
config_api_svcmonitor_1
config_api_nodemgr_1
config_api_dnsmasq_1
config_api_schema_1
2024-11-04 06:40:11,753 - DEBUG - [10.0.0.24]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}'
2024-11-04 06:40:11,834 - DEBUG - Output : 10.0.0.24/24
172.17.0.1/16
2024-11-04 06:40:11,834 - DEBUG - [10.0.0.24]: Running cmd : getent hosts 10.0.0.24 | head -n 1 | awk '{print $2}'
2024-11-04 06:40:11,868 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-759-1
2024-11-04 06:40:11,969 - DEBUG - [10.0.0.43]: Running cmd : hostname
2024-11-04 06:40:12,174 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-759-1
2024-11-04 06:40:12,174 - DEBUG - [10.0.0.43]: Running cmd : hostname -f
2024-11-04 06:40:12,208 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-759-1
2024-11-04 06:40:12,208 - DEBUG - [10.0.0.43]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2024-11-04 06:40:12,252 - DEBUG - Output : NAMES
vrouter_nodemgr_1
vrouter_vrouter-agent_1
2024-11-04 06:40:12,253 - DEBUG - [10.0.0.43]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}'
2024-11-04 06:40:12,325 - DEBUG - Output : 10.0.0.43/24
2024-11-04 06:40:12,326 - DEBUG - [10.0.0.43]: Running cmd : getent hosts 10.0.0.43 | head -n 1 | awk '{print $2}'
2024-11-04 06:40:12,360 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-759-1
2024-11-04 06:40:12,440 - INFO - Using existing project ['default-domain', 'k8s-default'](e94ef266-d145-4666-8a77-c4ef379f6cce)
2024-11-04 06:40:12,558 - INFO - Using existing project ['default-domain', 'k8s-default'](e94ef266-d145-4666-8a77-c4ef379f6cce)
2024-11-04 06:40:12,693 - INFO - ================================================================================
2024-11-04 06:40:12,693 - INFO - STARTING TEST : test_ingress_ip_assignment
2024-11-04 06:40:12,694 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2024-11-04 06:40:12,924 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_AgentXmppConnectionStatusReq?
2024-11-04 06:40:12,931 - DEBUG - Nothing to compare xmpp stats {'10.0.0.43': {'10.0.0.24': '0'}} with
2024-11-04 06:40:12,931 - INFO - Initial checks done. Running the testcase now
2024-11-04 06:40:12,931 - INFO -
2024-11-04 06:40:12,935 - INFO - Namespace default already exists
2024-11-04 06:40:12,935 - DEBUG - [10.0.0.24]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2024-11-04 06:40:13,041 - DEBUG - Output : cluster_project={}
2024-11-04 06:40:13,100 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2024-11-04 06:40:13,100 - DEBUG - Project uuid in Contrail is e94ef266-d145-4666-8a77-c4ef379f6cce
2024-11-04 06:40:13,100 - DEBUG - [10.0.0.24]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2024-11-04 06:40:13,229 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2024-11-04 06:40:13,230 - DEBUG - Requesting: http://10.0.0.24:8108/Snh_NamespaceDatabaseList?namespace_uuid=9cec0c06-bc49-424d-a758-ddedb6a6c15a
2024-11-04 06:40:13,234 - INFO - Namespace default with uuid 9cec0c06-bc49-424d-a758-ddedb6a6c15a found in kube manager
2024-11-04 06:40:13,234 - INFO - Namespace default verification passed
2024-11-04 06:40:13,237 - DEBUG - Service ctest-nginx-svc-67335053 not present
2024-11-04 06:40:13,238 - INFO - Creating service ctest-nginx-svc-67335053
2024-11-04 06:40:13,261 - INFO - Created Service ctest-nginx-svc-67335053
2024-11-04 06:40:13,278 - DEBUG - Pod ctest-nginx-pod-95459909 not present
2024-11-04 06:40:13,280 - INFO - Creating Pod ctest-nginx-pod-95459909
2024-11-04 06:40:13,352 - DEBUG - Pod : ctest-nginx-pod-95459909 UUID is 1101f5c2-6e72-46b0-a85f-cae3340f8ddd
2024-11-04 06:40:13,355 - DEBUG - Pod ctest-nginx-pod-61676120 not present
2024-11-04 06:40:13,356 - INFO - Creating Pod ctest-nginx-pod-61676120
2024-11-04 06:40:13,384 - DEBUG - Pod : ctest-nginx-pod-61676120 UUID is 6e068ca9-0a87-47f8-b75c-3ce8e8093270
2024-11-04 06:40:13,582 - INFO - Created VN __public__, UUID :7fad155d-654d-4668-b543-5c3d28fa4d66
2024-11-04 06:40:13,859 - DEBUG - Requesting: http://10.0.0.24:8082/domains
2024-11-04 06:40:13,866 - DEBUG - Requesting: http://10.0.0.24:8082/domain/d8e2c2e1-7015-4c51-b800-7b81e9d92551
2024-11-04 06:40:13,885 - DEBUG - Requesting: http://10.0.0.24:8082/project/e94ef266-d145-4666-8a77-c4ef379f6cce
2024-11-04 06:40:13,939 - DEBUG - Requesting: http://10.0.0.24:8082/virtual-network/7fad155d-654d-4668-b543-5c3d28fa4d66
2024-11-04 06:40:13,957 - DEBUG - Requesting: http://10.0.0.24:8082/virtual-network/7fad155d-654d-4668-b543-5c3d28fa4d66
2024-11-04 06:40:13,971 - DEBUG - Requesting: http://10.0.0.24:8082/routing-instance/0e16d189-6cda-49fd-a95a-2cdaa916aba1
2024-11-04 06:40:14,005 - DEBUG - Requesting: http://10.0.0.24:8082/routing-instance/0e16d189-6cda-49fd-a95a-2cdaa916aba1
2024-11-04 06:40:14,017 - DEBUG - Requesting: http://10.0.0.24:8082/route-target/e9a70b8d-c66f-4ad9-929a-2b2016497e07
2024-11-04 06:40:14,024 - DEBUG - Route Targets: ['target:64512:8000007']
2024-11-04 06:40:14,024 - DEBUG - Requesting: http://10.0.0.24:8082/virtual-network/7fad155d-654d-4668-b543-5c3d28fa4d66
2024-11-04 06:40:14,038 - DEBUG - Requesting: http://10.0.0.24:8082/routing-instance/0e16d189-6cda-49fd-a95a-2cdaa916aba1
2024-11-04 06:40:14,058 - INFO - Verified VN network id 13 for VN 7fad155d-654d-4668-b543-5c3d28fa4d66
2024-11-04 06:40:14,058 - INFO - Verifications in API Server for VN __public__ passed
2024-11-04 06:40:14,058 - DEBUG - Requesting: http://10.0.0.24:8082/virtual-network/7fad155d-654d-4668-b543-5c3d28fa4d66
2024-11-04 06:40:14,071 - DEBUG - Requesting: http://10.0.0.24:8082/routing-instance/0e16d189-6cda-49fd-a95a-2cdaa916aba1
2024-11-04 06:40:14,110 - DEBUG - Requesting: http://10.0.0.24:8082/routing-instance/0e16d189-6cda-49fd-a95a-2cdaa916aba1
2024-11-04 06:40:14,142 - DEBUG - Requesting: http://10.0.0.24:8082/route-target/e9a70b8d-c66f-4ad9-929a-2b2016497e07
2024-11-04 06:40:14,156 - DEBUG - Requesting: http://10.0.0.24:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2024-11-04 06:40:14,167 - DEBUG - Control-node 10.0.0.24 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9200033104663496296', 'uuid-lslong': '13061384762054823270'}, 'enable': 'true', 'created': '2024-11-04T06:40:13', 'last-modified': '2024-11-04T06:40:13', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.465284'}
2024-11-04 06:40:14,167 - DEBUG - Requesting: http://10.0.0.24:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2024-11-04 06:40:14,208 - DEBUG - Route Targets: ['target:64512:8000007']
2024-11-04 06:40:14,208 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2024-11-04 06:40:14,211 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2024-11-04 06:40:14,212 - DEBUG - Requesting: http://10.0.0.24:8082/domains
2024-11-04 06:40:14,230 - DEBUG - Requesting: http://10.0.0.24:8082/domain/d8e2c2e1-7015-4c51-b800-7b81e9d92551
2024-11-04 06:40:14,244 - DEBUG - Requesting: http://10.0.0.24:8082/project/e94ef266-d145-4666-8a77-c4ef379f6cce
2024-11-04 06:40:14,367 - DEBUG - Requesting: http://10.0.0.24:8082/virtual-network/7fad155d-654d-4668-b543-5c3d28fa4d66
2024-11-04 06:40:14,382 - DEBUG - =>VN __public__ has no policy to be verified
2024-11-04 06:40:14,382 - DEBUG - Verifying the vn in opserver
2024-11-04 06:40:14,382 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.24
2024-11-04 06:40:14,382 - DEBUG - Requesting: http://10.0.0.24:8081/analytics/uves/virtual-networks
2024-11-04 06:40:14,388 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.24:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2024-11-04 06:40:14,388 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2024-11-04 06:40:14,389 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2024-11-04 06:40:14,396 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2024-11-04 06:40:14,411 - DEBUG - Do not have enough data to verify VN in agent
2024-11-04 06:40:14,412 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2024-11-04 06:40:14,420 - DEBUG - VRF ids for VN __public__: {}
2024-11-04 06:40:14,564 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2024-11-04 06:40:14,684 - DEBUG - Requesting: http://10.0.0.24:8082/domains
2024-11-04 06:40:14,694 - DEBUG - Requesting: http://10.0.0.24:8082/domain/d8e2c2e1-7015-4c51-b800-7b81e9d92551
2024-11-04 06:40:14,706 - DEBUG - Requesting: http://10.0.0.24:8082/project/e94ef266-d145-4666-8a77-c4ef379f6cce
2024-11-04 06:40:14,756 - DEBUG - Requesting: http://10.0.0.24:8082/virtual-network/7fad155d-654d-4668-b543-5c3d28fa4d66
2024-11-04 06:40:14,773 - DEBUG - Requesting: http://10.0.0.24:8082/floating-ip-pool/f9c79d99-9a19-4661-acd9-53ad04155205
2024-11-04 06:40:14,783 - DEBUG - Requesting: http://10.0.0.24:8082/domains
2024-11-04 06:40:14,802 - DEBUG - Requesting: http://10.0.0.24:8082/domain/d8e2c2e1-7015-4c51-b800-7b81e9d92551
2024-11-04 06:40:14,811 - DEBUG - Requesting: http://10.0.0.24:8082/project/e94ef266-d145-4666-8a77-c4ef379f6cce
2024-11-04 06:40:14,884 - DEBUG - Requesting: http://10.0.0.24:8082/virtual-network/7fad155d-654d-4668-b543-5c3d28fa4d66
2024-11-04 06:40:14,897 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2024-11-04 06:40:14,897 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2024-11-04 06:40:14,937 - DEBUG - Requesting: http://10.0.0.24:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2024-11-04 06:40:14,950 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2024-11-04 06:40:14,950 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2024-11-04 06:40:14,958 - DEBUG - Ingress ctest-nginx-ingress-35602093 not present
2024-11-04 06:40:14,959 - INFO - Creating Ingress ctest-nginx-ingress-35602093
2024-11-04 06:40:14,969 - INFO - Created Ingress ctest-nginx-ingress-35602093
2024-11-04 06:40:14,979 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-35602093
2024-11-04 06:40:17,984 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-35602093
2024-11-04 06:40:20,993 - DEBUG - For Ingress ctest-nginx-ingress-35602093, Cluster IP: 10.47.255.249, External IPs ['9.242.197.252']
2024-11-04 06:40:20,994 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-35602093
2024-11-04 06:40:20,994 - DEBUG - Requesting: http://10.0.0.24:8108/Snh_LoadbalancerDatabaseList?x=f446f7cf-81a4-4496-8b92-4b4a6d3d3b8a
2024-11-04 06:40:20,999 - INFO - Ingress ctest-nginx-ingress-35602093 with uuid f446f7cf-81a4-4496-8b92-4b4a6d3d3b8a found in kube manager
2024-11-04 06:40:21,003 - INFO - Ingress ctest-nginx-ingress-35602093 verification passed
2024-11-04 06:40:21,006 - DEBUG - Pod ctest-busybox-pod-50801269 not present
2024-11-04 06:40:21,006 - INFO - Creating Pod ctest-busybox-pod-50801269
2024-11-04 06:40:21,029 - DEBUG - Pod : ctest-busybox-pod-50801269 UUID is 99bbb464-9856-4512-aaab-8b6891c48020
2024-11-04 06:40:21,040 - DEBUG - Pod ctest-nginx-pod-95459909 not in running state.Currently in Pending
2024-11-04 06:40:26,049 - INFO - Pod ctest-nginx-pod-95459909 is in running state.Got IP 10.47.255.251
2024-11-04 06:40:26,098 - DEBUG - Pod ctest-nginx-pod-95459909 has vmi a64a55b6-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:40:26,098 - INFO - Verified pod ctest-nginx-pod-95459909 in contrail-api
2024-11-04 06:40:26,098 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:40:26,127 - DEBUG - VMI a64a55b6-9a77-11ef-a6a2-fa163eef28a2 is active in agent 10.0.0.43
2024-11-04 06:40:26,127 - INFO - Verified Pod ctest-nginx-pod-95459909 in agent 10.0.0.43
2024-11-04 06:40:26,128 - DEBUG - Requesting: http://10.0.0.24:8108/Snh_VirtualMachineDatabaseList?x=1101f5c2-6e72-46b0-a85f-cae3340f8ddd
2024-11-04 06:40:26,136 - INFO - Pod ctest-nginx-pod-95459909 with uuid 1101f5c2-6e72-46b0-a85f-cae3340f8ddd found in kube manager
2024-11-04 06:40:26,136 - INFO - Pod ctest-nginx-pod-95459909 verification passed
2024-11-04 06:40:26,224 - DEBUG - [Pod ctest-nginx-pod-95459909] Cmd: echo ctest-nginx-pod-95459909 > /usr/share/nginx/html/index.html, Output:
2024-11-04 06:40:26,232 - DEBUG - Pod ctest-nginx-pod-61676120 not in running state.Currently in Pending
2024-11-04 06:40:31,238 - INFO - Pod ctest-nginx-pod-61676120 is in running state.Got IP 10.47.255.250
2024-11-04 06:40:31,316 - DEBUG - Pod ctest-nginx-pod-61676120 has vmi a6cabcd8-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:40:31,316 - INFO - Verified pod ctest-nginx-pod-61676120 in contrail-api
2024-11-04 06:40:31,316 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:40:31,346 - DEBUG - VMI a6cabcd8-9a77-11ef-a6a2-fa163eef28a2 is active in agent 10.0.0.43
2024-11-04 06:40:31,346 - INFO - Verified Pod ctest-nginx-pod-61676120 in agent 10.0.0.43
2024-11-04 06:40:31,351 - DEBUG - Requesting: http://10.0.0.24:8108/Snh_VirtualMachineDatabaseList?x=6e068ca9-0a87-47f8-b75c-3ce8e8093270
2024-11-04 06:40:31,368 - INFO - Pod ctest-nginx-pod-61676120 with uuid 6e068ca9-0a87-47f8-b75c-3ce8e8093270 found in kube manager
2024-11-04 06:40:31,368 - INFO - Pod ctest-nginx-pod-61676120 verification passed
2024-11-04 06:40:31,452 - DEBUG - [Pod ctest-nginx-pod-61676120] Cmd: echo ctest-nginx-pod-61676120 > /usr/share/nginx/html/index.html, Output:
2024-11-04 06:40:31,463 - DEBUG - Pod ctest-busybox-pod-50801269 not in running state.Currently in Pending
2024-11-04 06:40:36,471 - INFO - Pod ctest-busybox-pod-50801269 is in running state.Got IP 10.47.255.248
2024-11-04 06:40:36,491 - DEBUG - Pod ctest-busybox-pod-50801269 has vmi a9d4a84e-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:40:36,491 - INFO - Verified pod ctest-busybox-pod-50801269 in contrail-api
2024-11-04 06:40:36,491 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:40:36,502 - DEBUG - VMI a9d4a84e-9a77-11ef-a6a2-fa163eef28a2 is active in agent 10.0.0.43
2024-11-04 06:40:36,502 - INFO - Verified Pod ctest-busybox-pod-50801269 in agent 10.0.0.43
2024-11-04 06:40:36,502 - DEBUG - Requesting: http://10.0.0.24:8108/Snh_VirtualMachineDatabaseList?x=99bbb464-9856-4512-aaab-8b6891c48020
2024-11-04 06:40:36,507 - INFO - Pod ctest-busybox-pod-50801269 with uuid 99bbb464-9856-4512-aaab-8b6891c48020 found in kube manager
2024-11-04 06:40:36,507 - INFO - Pod ctest-busybox-pod-50801269 verification passed
2024-11-04 06:40:36,584 - DEBUG - [Pod ctest-busybox-pod-50801269] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-61676120
2024-11-04 06:40:36,584 - DEBUG - [Pod Pod ctest-busybox-pod-50801269] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2024-11-04 06:40:36,584 - DEBUG - [Pod Pod ctest-busybox-pod-50801269] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-61676120
2024-11-04 06:40:36,657 - DEBUG - [Pod ctest-busybox-pod-50801269] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-95459909
2024-11-04 06:40:36,657 - DEBUG - [Pod Pod ctest-busybox-pod-50801269] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2024-11-04 06:40:36,657 - DEBUG - [Pod Pod ctest-busybox-pod-50801269] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-95459909
2024-11-04 06:40:36,657 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-95459909': 1, 'ctest-nginx-pod-61676120': 1}
2024-11-04 06:40:36,658 - WARNING - Nothing to delete parallely
2024-11-04 06:40:36,658 - INFO - Deleting pod default:ctest-busybox-pod-50801269
2024-11-04 06:40:36,674 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=99bbb464-9856-4512-aaab-8b6891c48020
2024-11-04 06:40:36,689 - WARNING - Pod uuid 99bbb464-9856-4512-aaab-8b6891c48020 is still seen in agent 10.0.0.43 VM list
2024-11-04 06:40:41,690 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=99bbb464-9856-4512-aaab-8b6891c48020
2024-11-04 06:40:41,698 - WARNING - Pod uuid 99bbb464-9856-4512-aaab-8b6891c48020 is still seen in agent 10.0.0.43 VM list
2024-11-04 06:40:46,699 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=99bbb464-9856-4512-aaab-8b6891c48020
2024-11-04 06:40:46,714 - WARNING - Pod uuid 99bbb464-9856-4512-aaab-8b6891c48020 is still seen in agent 10.0.0.43 VM list
2024-11-04 06:40:51,715 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=99bbb464-9856-4512-aaab-8b6891c48020
2024-11-04 06:40:51,723 - WARNING - Pod uuid 99bbb464-9856-4512-aaab-8b6891c48020 is still seen in agent 10.0.0.43 VM list
2024-11-04 06:40:56,727 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=99bbb464-9856-4512-aaab-8b6891c48020
2024-11-04 06:40:56,734 - WARNING - Pod uuid 99bbb464-9856-4512-aaab-8b6891c48020 is still seen in agent 10.0.0.43 VM list
2024-11-04 06:41:01,735 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=99bbb464-9856-4512-aaab-8b6891c48020
2024-11-04 06:41:01,743 - WARNING - Pod uuid 99bbb464-9856-4512-aaab-8b6891c48020 is still seen in agent 10.0.0.43 VM list
2024-11-04 06:41:06,744 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=99bbb464-9856-4512-aaab-8b6891c48020
2024-11-04 06:41:06,751 - WARNING - Pod uuid 99bbb464-9856-4512-aaab-8b6891c48020 is still seen in agent 10.0.0.43 VM list
2024-11-04 06:41:11,752 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=99bbb464-9856-4512-aaab-8b6891c48020
2024-11-04 06:41:11,759 - DEBUG - Pod 99bbb464-9856-4512-aaab-8b6891c48020 is not in agent 10.0.0.43 VM list
2024-11-04 06:41:11,759 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a9d4a84e-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:41:11,765 - DEBUG - VMI a9d4a84e-9a77-11ef-a6a2-fa163eef28a2 is removed from agent 10.0.0.43
2024-11-04 06:41:11,765 - INFO - Verified that pod ctest-busybox-pod-50801269 is removed in agent
2024-11-04 06:41:11,766 - INFO - Deleting Ingress : ctest-nginx-ingress-35602093
2024-11-04 06:41:11,774 - DEBUG - Requesting: http://10.0.0.24:8108/Snh_LoadbalancerDatabaseList?x=f446f7cf-81a4-4496-8b92-4b4a6d3d3b8a
2024-11-04 06:41:11,781 - ERROR - Ingress ctest-nginx-ingress-35602093 with uuid f446f7cf-81a4-4496-8b92-4b4a6d3d3b8a still found in kube manager
2024-11-04 06:41:12,783 - DEBUG - Requesting: http://10.0.0.24:8108/Snh_LoadbalancerDatabaseList?x=f446f7cf-81a4-4496-8b92-4b4a6d3d3b8a
2024-11-04 06:41:12,789 - INFO - Ingress ctest-nginx-ingress-35602093 with uuid f446f7cf-81a4-4496-8b92-4b4a6d3d3b8a deleted successfully from kube manager
2024-11-04 06:41:12,789 - INFO - Verifications on Ingress %s cleanup passed
2024-11-04 06:41:12,789 - INFO - Deleting the FIP pool __fip_pool_public__
2024-11-04 06:41:12,859 - DEBUG - Requesting: http://10.0.0.24:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2024-11-04 06:41:12,865 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2024-11-04 06:41:12,865 - INFO - Deleting VN __public__
2024-11-04 06:41:12,996 - DEBUG - Requesting: http://10.0.0.24:8082/routing-instance/0e16d189-6cda-49fd-a95a-2cdaa916aba1
2024-11-04 06:41:13,009 - DEBUG - Response Code: 404
2024-11-04 06:41:13,009 - DEBUG - Requesting: http://10.0.0.24:8082/domains
2024-11-04 06:41:13,016 - DEBUG - Requesting: http://10.0.0.24:8082/domain/d8e2c2e1-7015-4c51-b800-7b81e9d92551
2024-11-04 06:41:13,031 - DEBUG - Requesting: http://10.0.0.24:8082/project/e94ef266-d145-4666-8a77-c4ef379f6cce
2024-11-04 06:41:13,080 - INFO - Validated that VN __public__ is not found in API Server
2024-11-04 06:41:13,080 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2024-11-04 06:41:13,099 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2024-11-04 06:41:13,118 - DEBUG - VN __public__ is not present in Agent 10.0.0.43
2024-11-04 06:41:13,118 - INFO - Validated that VN __public__ is not in any agent
2024-11-04 06:41:13,118 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2024-11-04 06:41:13,127 - DEBUG - Requesting: http://10.0.0.24:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2024-11-04 06:41:13,139 - DEBUG - Requesting: http://10.0.0.24:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2024-11-04 06:41:13,145 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2024-11-04 06:41:13,145 - INFO - Deleting pod default:ctest-nginx-pod-61676120
2024-11-04 06:41:13,164 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=6e068ca9-0a87-47f8-b75c-3ce8e8093270
2024-11-04 06:41:13,190 - WARNING - Pod uuid 6e068ca9-0a87-47f8-b75c-3ce8e8093270 is still seen in agent 10.0.0.43 VM list
2024-11-04 06:41:18,191 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=6e068ca9-0a87-47f8-b75c-3ce8e8093270
2024-11-04 06:41:18,198 - DEBUG - Pod 6e068ca9-0a87-47f8-b75c-3ce8e8093270 is not in agent 10.0.0.43 VM list
2024-11-04 06:41:18,198 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a6cabcd8-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:41:18,204 - DEBUG - VMI a6cabcd8-9a77-11ef-a6a2-fa163eef28a2 is removed from agent 10.0.0.43
2024-11-04 06:41:18,204 - INFO - Verified that pod ctest-nginx-pod-61676120 is removed in agent
2024-11-04 06:41:18,205 - INFO - Deleting pod default:ctest-nginx-pod-95459909
2024-11-04 06:41:18,216 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=1101f5c2-6e72-46b0-a85f-cae3340f8ddd
2024-11-04 06:41:18,224 - WARNING - Pod uuid 1101f5c2-6e72-46b0-a85f-cae3340f8ddd is still seen in agent 10.0.0.43 VM list
2024-11-04 06:41:23,225 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=1101f5c2-6e72-46b0-a85f-cae3340f8ddd
2024-11-04 06:41:23,233 - DEBUG - Pod 1101f5c2-6e72-46b0-a85f-cae3340f8ddd is not in agent 10.0.0.43 VM list
2024-11-04 06:41:23,233 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a64a55b6-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:41:23,239 - DEBUG - VMI a64a55b6-9a77-11ef-a6a2-fa163eef28a2 is removed from agent 10.0.0.43
2024-11-04 06:41:23,239 - INFO - Verified that pod ctest-nginx-pod-95459909 is removed in agent
2024-11-04 06:41:23,239 - INFO - Deleting service : ctest-nginx-svc-67335053
2024-11-04 06:41:23,446 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_AgentXmppConnectionStatusReq?
2024-11-04 06:41:23,453 - DEBUG - No XMPP flaps were noticed during the test
2024-11-04 06:41:23,453 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:11]
2024-11-04 06:41:23,453 - INFO - --------------------------------------------------------------------------------
2024-11-04 06:41:29,829 - DEBUG - [10.0.0.24]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2024-11-04 06:41:29,958 - DEBUG - Output : cluster_project={}
2024-11-04 06:41:30,013 - DEBUG - [10.0.0.24]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2024-11-04 06:41:30,140 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2024-11-04 06:41:30,145 - DEBUG - [10.0.0.24]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2024-11-04 06:41:30,302 - DEBUG - Output : cluster_project={}
2024-11-04 06:41:45,402 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:41:45,488 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:41:45,529 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:41:45,620 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:41:50,703 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:41:50,832 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:41:50,958 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:41:51,071 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-04 06:41:51,365 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_AgentXmppConnectionStatusReq?
2024-11-04 06:41:54,166 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_ApplicationPolicySetReq?
2024-11-04 06:41:54,173 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_AclReq?
2024-11-04 06:41:54,232 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_AclReq?
2024-11-04 06:41:56,296 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_AclReq?
2024-11-04 06:44:56,866 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_AgentXmppConnectionStatusReq?
2024-11-04 06:44:56,914 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=fae101c8-e46c-465c-8c1f-8a217464280e
2024-11-04 06:44:56,940 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=0e34846f-c5ef-4b2a-a420-35adac9cb4c8
2024-11-04 06:44:56,947 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=5d945252-09b0-46f9-8830-c074907ed653
2024-11-04 06:44:56,954 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=055ee4e4-37d4-4588-9c29-5c9f704b55f8
2024-11-04 06:44:56,965 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=12aef489-2ebb-46ed-86a3-191278f1b696
2024-11-04 06:44:56,969 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=00db8505-fb0e-49e1-941a-f490ce24bf56
2024-11-04 06:44:56,977 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=2a62e95a-74df-4391-bfdd-9a915b5db64e
2024-11-04 06:44:56,982 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=1b0fe9cd-cfb2-4e00-bf03-467118f1e810
2024-11-04 06:45:02,044 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=5d945252-09b0-46f9-8830-c074907ed653
2024-11-04 06:45:02,045 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=0e34846f-c5ef-4b2a-a420-35adac9cb4c8
2024-11-04 06:45:02,047 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=fae101c8-e46c-465c-8c1f-8a217464280e
2024-11-04 06:45:02,049 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=055ee4e4-37d4-4588-9c29-5c9f704b55f8
2024-11-04 06:45:02,050 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=2a62e95a-74df-4391-bfdd-9a915b5db64e
2024-11-04 06:45:02,051 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=1b0fe9cd-cfb2-4e00-bf03-467118f1e810
2024-11-04 06:45:02,052 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=12aef489-2ebb-46ed-86a3-191278f1b696
2024-11-04 06:45:02,053 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=00db8505-fb0e-49e1-941a-f490ce24bf56
2024-11-04 06:45:02,077 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d3c73d06-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:45:02,082 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d5a545aa-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:45:02,084 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d50b36ea-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:45:02,086 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d451361e-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:45:07,078 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=055ee4e4-37d4-4588-9c29-5c9f704b55f8
2024-11-04 06:45:07,085 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=1b0fe9cd-cfb2-4e00-bf03-467118f1e810
2024-11-04 06:45:07,086 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=2a62e95a-74df-4391-bfdd-9a915b5db64e
2024-11-04 06:45:07,087 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=5d945252-09b0-46f9-8830-c074907ed653
2024-11-04 06:45:12,086 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=055ee4e4-37d4-4588-9c29-5c9f704b55f8
2024-11-04 06:45:12,101 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=5d945252-09b0-46f9-8830-c074907ed653
2024-11-04 06:45:12,103 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=2a62e95a-74df-4391-bfdd-9a915b5db64e
2024-11-04 06:45:12,104 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=1b0fe9cd-cfb2-4e00-bf03-467118f1e810
2024-11-04 06:45:17,093 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=055ee4e4-37d4-4588-9c29-5c9f704b55f8
2024-11-04 06:45:17,116 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=5d945252-09b0-46f9-8830-c074907ed653
2024-11-04 06:45:17,117 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=2a62e95a-74df-4391-bfdd-9a915b5db64e
2024-11-04 06:45:17,119 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=1b0fe9cd-cfb2-4e00-bf03-467118f1e810
2024-11-04 06:45:22,101 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=055ee4e4-37d4-4588-9c29-5c9f704b55f8
2024-11-04 06:45:22,131 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=2a62e95a-74df-4391-bfdd-9a915b5db64e
2024-11-04 06:45:22,132 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=1b0fe9cd-cfb2-4e00-bf03-467118f1e810
2024-11-04 06:45:22,134 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=5d945252-09b0-46f9-8830-c074907ed653
2024-11-04 06:45:27,111 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=055ee4e4-37d4-4588-9c29-5c9f704b55f8
2024-11-04 06:45:27,148 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=2a62e95a-74df-4391-bfdd-9a915b5db64e
2024-11-04 06:45:27,150 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=1b0fe9cd-cfb2-4e00-bf03-467118f1e810
2024-11-04 06:45:27,151 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=5d945252-09b0-46f9-8830-c074907ed653
2024-11-04 06:45:32,136 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=055ee4e4-37d4-4588-9c29-5c9f704b55f8
2024-11-04 06:45:32,144 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d6401990-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:45:32,167 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=5d945252-09b0-46f9-8830-c074907ed653
2024-11-04 06:45:32,168 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=2a62e95a-74df-4391-bfdd-9a915b5db64e
2024-11-04 06:45:32,171 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_VmListReq?uuid=1b0fe9cd-cfb2-4e00-bf03-467118f1e810
2024-11-04 06:45:32,183 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d6b65fec-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:45:32,185 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d7c5b61c-9a77-11ef-a6a2-fa163eef28a2
2024-11-04 06:45:32,195 - DEBUG - Requesting: http://10.0.0.43:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d73e4db2-9a77-11ef-a6a2-fa163eef28a2