2025-02-10 10:09:25,020 - DEBUG - [10.0.0.29]: Running cmd : hostname
2025-02-10 10:09:25,318 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-290-1.novalocal
2025-02-10 10:09:25,319 - DEBUG - [10.0.0.29]: Running cmd : hostname -f
2025-02-10 10:09:25,458 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-290-1.novalocal
2025-02-10 10:09:25,458 - DEBUG - [10.0.0.29]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-02-10 10:09:25,631 - DEBUG - Output : NAMES
contrail_test_HquzGFszI
vrouter_vrouter-agent_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
kubemanager_k8s_kubemanager_1
analytics_snmp_snmp-collector_1
analytics_snmp_nodemgr_1
analytics_snmp_topology_1
analytics_alarm_nodemgr_1
analytics_alarm_alarm-gen_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_collector_1
analytics_api_1
analytics_database_query-engine_1
analytics_database_nodemgr_1
analytics_database_cassandra_1
control_named_1
control_dns_1
control_nodemgr_1
control_control_1
webui_job_1
webui_web_1
config_devicemgr_1
config_svcmonitor_1
config_api_1
config_dnsmasq_1
config_schema_1
config_nodemgr_1
config_database_rabbitmq_1
config_database_nodemgr_1
config_database_cassandra_1
config_database_zookeeper_1
redis_redis_1
2025-02-10 10:09:25,631 - DEBUG - [10.0.0.29]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-02-10 10:09:25,763 - DEBUG - Output : 172.17.0.1/16
10.0.0.29/24
2025-02-10 10:09:25,763 - DEBUG - [10.0.0.29]: Running cmd : getent hosts 10.0.0.29 | head -n 1 | awk '{print $2}'
2025-02-10 10:09:25,911 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-290-1.novalocal
2025-02-10 10:09:25,911 - DEBUG - [10.0.0.29]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-02-10 10:09:26,039 - DEBUG - Output : 10.0.0.29/24
2025-02-10 10:09:26,316 - INFO - Using existing project ['default-domain', 'k8s-default'](1aa56c47-e077-4797-b714-17dc2a84822d)
2025-02-10 10:09:26,403 - INFO - Using existing project ['default-domain', 'k8s-default'](1aa56c47-e077-4797-b714-17dc2a84822d)
2025-02-10 10:09:26,573 - INFO - ================================================================================
2025-02-10 10:09:26,573 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-02-10 10:09:26,574 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-02-10 10:09:26,827 - DEBUG - Requesting: http://10.0.0.29:8085/Snh_AgentXmppConnectionStatusReq?
2025-02-10 10:09:26,837 - DEBUG - Nothing to compare xmpp stats {'10.0.0.29': {'10.0.0.29': '0'}} with
2025-02-10 10:09:26,837 - INFO - Initial checks done. Running the testcase now
2025-02-10 10:09:26,837 - INFO -
2025-02-10 10:09:26,842 - INFO - Namespace default already exists
2025-02-10 10:09:26,842 - DEBUG - [10.0.0.29]: Running cmd : docker exec --privileged -it kubemanager_k8s_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-02-10 10:09:27,039 - DEBUG - Output : cluster_project={}
2025-02-10 10:09:27,101 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-02-10 10:09:27,101 - DEBUG - Project uuid in Contrail is 1aa56c47-e077-4797-b714-17dc2a84822d
2025-02-10 10:09:27,102 - DEBUG - [10.0.0.29]: Running cmd : docker exec --privileged -it kubemanager_k8s_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-02-10 10:09:27,272 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-02-10 10:09:27,272 - DEBUG - Requesting: http://10.0.0.29:8108/Snh_NamespaceDatabaseList?namespace_uuid=6eb8efbf-83be-44a7-be6d-6a41cf2b0752
2025-02-10 10:09:27,277 - INFO - Namespace default with uuid 6eb8efbf-83be-44a7-be6d-6a41cf2b0752 found in kube manager
2025-02-10 10:09:27,277 - INFO - Namespace default verification passed
2025-02-10 10:09:27,280 - DEBUG - Service ctest-nginx-svc-72055328 not present
2025-02-10 10:09:27,280 - INFO - Creating service ctest-nginx-svc-72055328
2025-02-10 10:09:27,303 - INFO - Created Service ctest-nginx-svc-72055328
2025-02-10 10:09:27,307 - DEBUG - Pod ctest-nginx-pod-42836535 not present
2025-02-10 10:09:27,307 - INFO - Creating Pod ctest-nginx-pod-42836535
2025-02-10 10:09:27,369 - DEBUG - Pod : ctest-nginx-pod-42836535 UUID is 0cb20776-d632-4462-922c-f612f9ff5b39
2025-02-10 10:09:27,373 - DEBUG - Pod ctest-nginx-pod-95003398 not present
2025-02-10 10:09:27,374 - INFO - Creating Pod ctest-nginx-pod-95003398
2025-02-10 10:09:27,402 - DEBUG - Pod : ctest-nginx-pod-95003398 UUID is fa8456c2-0951-4651-8a07-2d51eaaf723e
2025-02-10 10:09:27,619 - INFO - Created VN __public__, UUID :e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:27,796 - DEBUG - Requesting: http://10.0.0.29:8082/domains
2025-02-10 10:09:27,803 - DEBUG - Requesting: http://10.0.0.29:8082/domain/544a2b6c-97bf-4a09-8358-45f8bec63190
2025-02-10 10:09:27,815 - DEBUG - Requesting: http://10.0.0.29:8082/project/1aa56c47-e077-4797-b714-17dc2a84822d
2025-02-10 10:09:27,866 - DEBUG - Requesting: http://10.0.0.29:8082/virtual-network/e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:27,881 - DEBUG - Requesting: http://10.0.0.29:8082/virtual-network/e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:27,891 - DEBUG - Requesting: http://10.0.0.29:8082/routing-instance/b0575b34-0831-4d94-bb41-2a37427aa1e0
2025-02-10 10:09:27,899 - DEBUG - Requesting: http://10.0.0.29:8082/routing-instance/b0575b34-0831-4d94-bb41-2a37427aa1e0
2025-02-10 10:09:27,908 - DEBUG - Route Targets: []
2025-02-10 10:09:27,908 - DEBUG - RT names not yet present for VN __public__
2025-02-10 10:09:32,909 - DEBUG - Requesting: http://10.0.0.29:8082/domains
2025-02-10 10:09:32,915 - DEBUG - Requesting: http://10.0.0.29:8082/domain/544a2b6c-97bf-4a09-8358-45f8bec63190
2025-02-10 10:09:32,926 - DEBUG - Requesting: http://10.0.0.29:8082/project/1aa56c47-e077-4797-b714-17dc2a84822d
2025-02-10 10:09:32,987 - DEBUG - Requesting: http://10.0.0.29:8082/virtual-network/e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:33,003 - DEBUG - Requesting: http://10.0.0.29:8082/virtual-network/e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:33,019 - DEBUG - Requesting: http://10.0.0.29:8082/routing-instance/b0575b34-0831-4d94-bb41-2a37427aa1e0
2025-02-10 10:09:33,029 - DEBUG - Requesting: http://10.0.0.29:8082/routing-instance/b0575b34-0831-4d94-bb41-2a37427aa1e0
2025-02-10 10:09:33,037 - DEBUG - Requesting: http://10.0.0.29:8082/route-target/5692c263-f046-4768-a75e-0388acce44aa
2025-02-10 10:09:33,046 - DEBUG - Route Targets: ['target:64512:8000006']
2025-02-10 10:09:33,047 - DEBUG - Requesting: http://10.0.0.29:8082/virtual-network/e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:33,059 - DEBUG - Requesting: http://10.0.0.29:8082/routing-instance/b0575b34-0831-4d94-bb41-2a37427aa1e0
2025-02-10 10:09:33,077 - INFO - Verified VN network id 15 for VN e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:33,078 - INFO - Verifications in API Server for VN __public__ passed
2025-02-10 10:09:33,078 - DEBUG - Requesting: http://10.0.0.29:8082/virtual-network/e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:33,093 - DEBUG - Requesting: http://10.0.0.29:8082/routing-instance/b0575b34-0831-4d94-bb41-2a37427aa1e0
2025-02-10 10:09:33,102 - DEBUG - Requesting: http://10.0.0.29:8082/routing-instance/b0575b34-0831-4d94-bb41-2a37427aa1e0
2025-02-10 10:09:33,110 - DEBUG - Requesting: http://10.0.0.29:8082/route-target/5692c263-f046-4768-a75e-0388acce44aa
2025-02-10 10:09:33,116 - DEBUG - Requesting: http://10.0.0.29:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-02-10 10:09:33,126 - DEBUG - Control-node 10.0.0.29 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16632949610374578548', 'uuid-lslong': '13117428117591041893'}, 'enable': 'true', 'created': '2025-02-10T10:09:27', 'last-modified': '2025-02-10T10:09:27', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.413465'}
2025-02-10 10:09:33,126 - DEBUG - Requesting: http://10.0.0.29:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-02-10 10:09:33,147 - DEBUG - Route Targets: ['target:64512:8000006']
2025-02-10 10:09:33,147 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-02-10 10:09:33,148 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-02-10 10:09:33,148 - DEBUG - Requesting: http://10.0.0.29:8082/domains
2025-02-10 10:09:33,172 - DEBUG - Requesting: http://10.0.0.29:8082/domain/544a2b6c-97bf-4a09-8358-45f8bec63190
2025-02-10 10:09:33,182 - DEBUG - Requesting: http://10.0.0.29:8082/project/1aa56c47-e077-4797-b714-17dc2a84822d
2025-02-10 10:09:33,268 - DEBUG - Requesting: http://10.0.0.29:8082/virtual-network/e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:33,285 - DEBUG - =>VN __public__ has no policy to be verified
2025-02-10 10:09:33,285 - DEBUG - Verifying the vn in opserver
2025-02-10 10:09:33,285 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.29
2025-02-10 10:09:33,285 - DEBUG - Requesting: http://10.0.0.29:8081/analytics/uves/virtual-networks
2025-02-10 10:09:33,293 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.29:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-02-10 10:09:33,293 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-02-10 10:09:33,293 - DEBUG - Requesting: http://10.0.0.29:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-02-10 10:09:33,302 - DEBUG - Requesting: http://10.0.0.29:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-02-10 10:09:33,310 - DEBUG - Do not have enough data to verify VN in agent
2025-02-10 10:09:33,310 - DEBUG - Requesting: http://10.0.0.29:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-02-10 10:09:33,318 - DEBUG - VRF ids for VN __public__: {}
2025-02-10 10:09:33,411 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-02-10 10:09:33,501 - DEBUG - Requesting: http://10.0.0.29:8082/domains
2025-02-10 10:09:33,508 - DEBUG - Requesting: http://10.0.0.29:8082/domain/544a2b6c-97bf-4a09-8358-45f8bec63190
2025-02-10 10:09:33,516 - DEBUG - Requesting: http://10.0.0.29:8082/project/1aa56c47-e077-4797-b714-17dc2a84822d
2025-02-10 10:09:33,599 - DEBUG - Requesting: http://10.0.0.29:8082/virtual-network/e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:33,642 - DEBUG - Requesting: http://10.0.0.29:8082/floating-ip-pool/e0dbab45-fd62-49cd-82c4-e3b6f47683a5
2025-02-10 10:09:33,649 - DEBUG - Requesting: http://10.0.0.29:8082/domains
2025-02-10 10:09:33,655 - DEBUG - Requesting: http://10.0.0.29:8082/domain/544a2b6c-97bf-4a09-8358-45f8bec63190
2025-02-10 10:09:33,666 - DEBUG - Requesting: http://10.0.0.29:8082/project/1aa56c47-e077-4797-b714-17dc2a84822d
2025-02-10 10:09:33,732 - DEBUG - Requesting: http://10.0.0.29:8082/virtual-network/e6d41b8b-a559-4174-b60a-775fb3bc2765
2025-02-10 10:09:33,745 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-02-10 10:09:33,745 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-02-10 10:09:33,754 - DEBUG - Requesting: http://10.0.0.29:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-02-10 10:09:33,760 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-02-10 10:09:33,760 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-02-10 10:09:33,766 - DEBUG - Ingress ctest-nginx-ingress-98622015 not present
2025-02-10 10:09:33,766 - INFO - Creating Ingress ctest-nginx-ingress-98622015
2025-02-10 10:09:33,776 - INFO - Created Ingress ctest-nginx-ingress-98622015
2025-02-10 10:09:33,780 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-98622015
2025-02-10 10:09:36,785 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-98622015
2025-02-10 10:09:39,792 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-98622015
2025-02-10 10:09:42,799 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-98622015
2025-02-10 10:09:45,805 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-98622015
2025-02-10 10:09:48,810 - DEBUG - For Ingress ctest-nginx-ingress-98622015, Cluster IP: 10.47.255.246, External IPs ['33.140.88.60']
2025-02-10 10:09:48,810 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-98622015
2025-02-10 10:09:48,810 - DEBUG - Requesting: http://10.0.0.29:8108/Snh_LoadbalancerDatabaseList?x=b26a08fe-8a56-4c62-812e-709a6106a6a5
2025-02-10 10:09:48,815 - INFO - Ingress ctest-nginx-ingress-98622015 with uuid b26a08fe-8a56-4c62-812e-709a6106a6a5 found in kube manager
2025-02-10 10:09:48,815 - INFO - Ingress ctest-nginx-ingress-98622015 verification passed
2025-02-10 10:09:48,818 - DEBUG - Pod ctest-busybox-pod-43635281 not present
2025-02-10 10:09:48,818 - INFO - Creating Pod ctest-busybox-pod-43635281
2025-02-10 10:09:48,837 - DEBUG - Pod : ctest-busybox-pod-43635281 UUID is 77267d16-cc0c-4840-a397-6d649c2c298f
2025-02-10 10:09:48,847 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:09:53,856 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:09:58,864 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:03,871 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:08,881 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:13,888 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:18,895 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:23,903 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:28,910 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:33,917 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:38,928 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:43,936 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:48,943 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:53,951 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:10:58,957 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:03,965 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:08,973 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:13,980 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:18,987 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:23,996 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:29,004 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:34,012 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:39,020 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:44,027 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:49,034 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:54,041 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:11:59,049 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:04,056 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:09,064 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:14,071 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:19,079 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:24,088 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:29,095 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:34,103 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:39,110 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:44,118 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:49,125 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:54,131 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:12:59,138 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:04,146 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:09,155 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:14,162 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:19,170 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:24,182 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:29,189 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:34,197 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:39,208 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:44,215 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:49,223 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:54,232 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:13:59,245 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:04,255 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:09,263 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:14,272 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:19,282 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:24,290 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:29,298 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:34,306 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:39,313 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:44,323 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:49,332 - DEBUG - Pod ctest-nginx-pod-42836535 not in running state.Currently in Pending
2025-02-10 10:14:49,332 - ERROR - Pod ctest-nginx-pod-42836535 is not in running state
2025-02-10 10:14:49,338 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:14:54,348 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:14:59,355 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:04,364 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:09,372 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:14,379 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:19,388 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:24,398 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:29,406 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:34,414 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:39,421 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:44,429 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:49,437 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:54,445 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:15:59,454 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:04,461 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:09,469 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:14,476 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:19,486 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:24,493 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:29,501 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:34,508 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:39,515 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:44,523 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:49,532 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:54,540 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:16:59,554 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:04,561 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:09,568 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:14,576 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:19,582 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:24,590 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:29,597 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:34,604 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:39,614 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:44,621 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:49,628 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:54,636 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:17:59,643 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:04,653 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:09,661 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:14,668 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:19,676 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:24,683 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:29,690 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:34,697 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:39,706 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:44,714 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:49,722 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:54,730 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:18:59,737 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:04,744 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:09,751 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:14,759 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:19,768 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:24,775 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:29,783 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:34,793 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:39,800 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:44,811 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:49,819 - DEBUG - Pod ctest-nginx-pod-95003398 not in running state.Currently in Pending
2025-02-10 10:19:49,819 - ERROR - Pod ctest-nginx-pod-95003398 is not in running state
2025-02-10 10:19:49,825 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:19:54,833 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:19:59,840 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:04,847 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:09,855 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:14,865 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:19,872 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:24,879 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:29,887 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:34,896 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:39,903 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:44,911 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:49,920 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:54,927 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:20:59,934 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:04,942 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:09,949 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:14,957 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:19,965 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:24,972 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:29,979 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:34,991 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:40,000 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:45,008 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:50,016 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:21:55,023 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:00,032 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:05,040 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:10,048 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:15,061 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:20,068 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:25,076 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:30,084 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:35,091 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:40,098 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:45,105 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:50,115 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:22:55,123 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:00,131 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:05,138 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:10,147 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:15,154 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:20,165 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:25,173 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:30,181 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:35,189 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:40,199 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:45,209 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:50,221 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:23:55,229 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:00,237 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:05,245 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:10,254 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:15,263 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:20,271 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:25,279 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:30,286 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:35,294 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:40,303 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:45,311 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:50,319 - DEBUG - Pod ctest-busybox-pod-43635281 not in running state.Currently in Pending
2025-02-10 10:24:50,319 - ERROR - Pod ctest-busybox-pod-43635281 is not in running state
2025-02-10 10:24:50,327 - WARNING - Nothing to delete parallely
2025-02-10 10:24:50,327 - INFO - Deleting pod default:ctest-busybox-pod-43635281
2025-02-10 10:24:50,345 - DEBUG - Pod ctest-busybox-pod-43635281 may not have launched at all..no need to check further in agent
2025-02-10 10:24:50,345 - INFO - Deleting Ingress : ctest-nginx-ingress-98622015
2025-02-10 10:24:50,353 - DEBUG - Requesting: http://10.0.0.29:8108/Snh_LoadbalancerDatabaseList?x=b26a08fe-8a56-4c62-812e-709a6106a6a5
2025-02-10 10:24:50,358 - ERROR - Ingress ctest-nginx-ingress-98622015 with uuid b26a08fe-8a56-4c62-812e-709a6106a6a5 still found in kube manager
2025-02-10 10:24:51,359 - DEBUG - Requesting: http://10.0.0.29:8108/Snh_LoadbalancerDatabaseList?x=b26a08fe-8a56-4c62-812e-709a6106a6a5
2025-02-10 10:24:51,365 - ERROR - Ingress ctest-nginx-ingress-98622015 with uuid b26a08fe-8a56-4c62-812e-709a6106a6a5 still found in kube manager
2025-02-10 10:24:52,367 - DEBUG - Requesting: http://10.0.0.29:8108/Snh_LoadbalancerDatabaseList?x=b26a08fe-8a56-4c62-812e-709a6106a6a5
2025-02-10 10:24:52,375 - ERROR - Ingress ctest-nginx-ingress-98622015 with uuid b26a08fe-8a56-4c62-812e-709a6106a6a5 still found in kube manager
2025-02-10 10:24:53,376 - DEBUG - Requesting: http://10.0.0.29:8108/Snh_LoadbalancerDatabaseList?x=b26a08fe-8a56-4c62-812e-709a6106a6a5
2025-02-10 10:24:53,384 - INFO - Ingress ctest-nginx-ingress-98622015 with uuid b26a08fe-8a56-4c62-812e-709a6106a6a5 deleted successfully from kube manager
2025-02-10 10:24:53,384 - INFO - Verifications on Ingress %s cleanup passed
2025-02-10 10:24:53,384 - INFO - Deleting the FIP pool __fip_pool_public__
2025-02-10 10:24:53,588 - DEBUG - Requesting: http://10.0.0.29:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-02-10 10:24:53,596 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-02-10 10:24:53,596 - INFO - Deleting VN __public__
2025-02-10 10:24:54,291 - DEBUG - Requesting: http://10.0.0.29:8082/routing-instance/b0575b34-0831-4d94-bb41-2a37427aa1e0
2025-02-10 10:24:54,298 - DEBUG - Response Code: 404
2025-02-10 10:24:54,298 - DEBUG - Requesting: http://10.0.0.29:8082/domains
2025-02-10 10:24:54,305 - DEBUG - Requesting: http://10.0.0.29:8082/domain/544a2b6c-97bf-4a09-8358-45f8bec63190
2025-02-10 10:24:54,328 - DEBUG - Requesting: http://10.0.0.29:8082/project/1aa56c47-e077-4797-b714-17dc2a84822d
2025-02-10 10:24:54,472 - INFO - Validated that VN __public__ is not found in API Server
2025-02-10 10:24:54,472 - DEBUG - Requesting: http://10.0.0.29:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-02-10 10:24:54,480 - DEBUG - Requesting: http://10.0.0.29:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-02-10 10:24:54,487 - DEBUG - VN __public__ is not present in Agent 10.0.0.29
2025-02-10 10:24:54,487 - INFO - Validated that VN __public__ is not in any agent
2025-02-10 10:24:54,488 - DEBUG - Requesting: http://10.0.0.29:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-02-10 10:24:54,496 - DEBUG - Requesting: http://10.0.0.29:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-02-10 10:24:54,502 - DEBUG - Requesting: http://10.0.0.29:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-02-10 10:24:54,508 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-02-10 10:24:54,508 - INFO - Deleting pod default:ctest-nginx-pod-95003398
2025-02-10 10:24:54,528 - DEBUG - Pod ctest-nginx-pod-95003398 may not have launched at all..no need to check further in agent
2025-02-10 10:24:54,528 - INFO - Deleting pod default:ctest-nginx-pod-42836535
2025-02-10 10:24:54,539 - DEBUG - Pod ctest-nginx-pod-42836535 may not have launched at all..no need to check further in agent
2025-02-10 10:24:54,539 - INFO - Deleting service : ctest-nginx-svc-72055328
2025-02-10 10:24:55,092 - ERROR - AssertionError
Python 3.6.8: /usr/bin/python3
Mon Feb 10 10:24:50 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/contrail-test/tcutils/wrappers.py in wrapper(self=