2026-03-24 16:28:06,291 - DEBUG - [10.0.0.50]: Running cmd : hostname 2026-03-24 16:28:06,543 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-384-1 2026-03-24 16:28:06,543 - DEBUG - [10.0.0.50]: Running cmd : hostname -f 2026-03-24 16:28:06,668 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-384-1.cluster.local 2026-03-24 16:28:06,668 - DEBUG - [10.0.0.50]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-03-24 16:28:06,828 - DEBUG - Output : NAMES contrail_test_Y40VFOuiL k8s_autoscaler_dns-autoscaler-595558c478-z5wn7_kube-system_76fbf137-f08c-485f-848b-decf55748db8_0 k8s_coredns_coredns-77f7cc69db-g4cws_kube-system_e7bca482-e639-4773-a702-1ad3914f6581_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-nvq5k_kube-system_fe18d1d6-730f-4b35-a6b2-9896cd958347_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-nvq5k_kube-system_fe18d1d6-730f-4b35-a6b2-9896cd958347_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-xb6mm_kube-system_d4537c33-af89-483b-9f0b-579efe3c0361_0 k8s_opensdn-controller-control-named_opensdn-controller-control-xb6mm_kube-system_d4537c33-af89-483b-9f0b-579efe3c0361_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-nvq5k_kube-system_fe18d1d6-730f-4b35-a6b2-9896cd958347_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-9sd2v_kube-system_be713129-4237-4cd7-afba-c1f2b7bed8f0_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-hf8np_kube-system_8fd2c1eb-969d-4976-adce-118c3f365d28_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-hf8np_kube-system_8fd2c1eb-969d-4976-adce-118c3f365d28_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-vs2kw_kube-system_52fd3fb2-1f08-4c8f-815c-0a54b3ea5b93_0 k8s_opensdn-analytics-collector_opensdn-analytics-vs2kw_kube-system_52fd3fb2-1f08-4c8f-815c-0a54b3ea5b93_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-2bkj5_kube-system_1f4f742e-b1f8-4701-9aeb-ed6ed6b1c28b_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-xb6mm_kube-system_d4537c33-af89-483b-9f0b-579efe3c0361_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-97mqs_kube-system_7bc46071-4423-4bea-8331-8f11ea7b3edc_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-97mqs_kube-system_7bc46071-4423-4bea-8331-8f11ea7b3edc_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-nvq5k_kube-system_fe18d1d6-730f-4b35-a6b2-9896cd958347_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-9sd2v_kube-system_be713129-4237-4cd7-afba-c1f2b7bed8f0_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-hf8np_kube-system_8fd2c1eb-969d-4976-adce-118c3f365d28_0 k8s_opensdn-analytics-api_opensdn-analytics-vs2kw_kube-system_52fd3fb2-1f08-4c8f-815c-0a54b3ea5b93_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-2bkj5_kube-system_1f4f742e-b1f8-4701-9aeb-ed6ed6b1c28b_0 k8s_opensdn-controller-control_opensdn-controller-control-xb6mm_kube-system_d4537c33-af89-483b-9f0b-579efe3c0361_0 k8s_kafka_opensdn-analytics-alarm-97mqs_kube-system_7bc46071-4423-4bea-8331-8f11ea7b3edc_0 k8s_opensdn-controller-config-api_opensdn-controller-config-nvq5k_kube-system_fe18d1d6-730f-4b35-a6b2-9896cd958347_0 k8s_redis_redis-9mxmj_kube-system_16a77c6d-dc5f-483a-ac87-d0ecb6f26a98_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-q9klg_kube-system_fa1808ec-3017-4928-b22d-93399554597b_0 k8s_opensdn-configdb_opensdn-configdb-q9klg_kube-system_fa1808ec-3017-4928-b22d-93399554597b_0 k8s_rabbitmq_rabbitmq-d2nww_kube-system_a2233cc0-55fa-44ed-8f36-639009ae8740_0 k8s_config-zookeeper_config-zookeeper-pkvwr_kube-system_de152145-dcf0-47a0-bf3f-452fd53a59c6_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-9sd2v_kube-system_be713129-4237-4cd7-afba-c1f2b7bed8f0_0 k8s_opensdn-agent-nodemgr_opensdn-agent-9hqml_kube-system_ff5e1962-82d4-4939-a5e6-bcff83cdd13f_0 k8s_opensdn-vrouter-agent_opensdn-agent-9hqml_kube-system_ff5e1962-82d4-4939-a5e6-bcff83cdd13f_0 k8s_opensdn-kube-manager_opensdn-kube-manager-bbjr9_kube-system_13254165-f876-4a37-893a-85a14e055fa0_0 k8s_kube-proxy_kube-proxy-wt9fn_kube-system_1db05b3c-b51f-4b3e-9fea-563f194600e4_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-384-1_kube-system_d03f6c029c4b5509dadc743cfd0182d8_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-384-1_kube-system_5c8a19796c279923265d554015c161f9_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-384-1_kube-system_55a2c1767eda832f250559340665d109_0 2026-03-24 16:28:06,828 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-03-24 16:28:06,960 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.3/32 10.96.0.1/32 10.0.0.50/24 2026-03-24 16:28:06,960 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.0.0.50 | head -n 1 | awk '{print $2}' 2026-03-24 16:28:07,125 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-384-1.cluster.local 2026-03-24 16:28:07,125 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-03-24 16:28:07,242 - DEBUG - Output : 10.0.0.50/24 2026-03-24 16:28:07,627 - INFO - Using existing project ['default-domain', 'k8s-default'](425dfa0f-076a-49fd-9d13-e1fbcb09c639) 2026-03-24 16:28:08,112 - INFO - Using existing project ['default-domain', 'k8s-default'](425dfa0f-076a-49fd-9d13-e1fbcb09c639) 2026-03-24 16:28:08,255 - INFO - ================================================================================ 2026-03-24 16:28:08,255 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-03-24 16:28:08,257 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-03-24 16:28:08,540 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-24 16:28:08,545 - DEBUG - Nothing to compare xmpp stats {'10.0.0.50': {'10.0.0.50': '3'}} with 2026-03-24 16:28:08,545 - INFO - Initial checks done. Running the testcase now 2026-03-24 16:28:08,545 - INFO - 2026-03-24 16:28:08,550 - INFO - Namespace default already exists 2026-03-24 16:28:08,550 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-bbjr9_kube-system_13254165-f876-4a37-893a-85a14e055fa0_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-03-24 16:28:08,720 - DEBUG - Output : cluster_project={} 2026-03-24 16:28:08,795 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-03-24 16:28:08,796 - DEBUG - Project uuid in Contrail is 425dfa0f-076a-49fd-9d13-e1fbcb09c639 2026-03-24 16:28:08,796 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-bbjr9_kube-system_13254165-f876-4a37-893a-85a14e055fa0_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-03-24 16:28:08,945 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-03-24 16:28:08,945 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_NamespaceDatabaseList?namespace_uuid=86336d52-ff97-41c6-9bf4-c1b2bb9cae5d 2026-03-24 16:28:08,949 - INFO - Namespace default with uuid 86336d52-ff97-41c6-9bf4-c1b2bb9cae5d found in kube manager 2026-03-24 16:28:08,949 - INFO - Namespace default verification passed 2026-03-24 16:28:08,952 - DEBUG - Service ctest-nginx-svc-56768026 not present 2026-03-24 16:28:08,953 - INFO - Creating service ctest-nginx-svc-56768026 2026-03-24 16:28:08,965 - INFO - Created Service ctest-nginx-svc-56768026 2026-03-24 16:28:08,970 - DEBUG - Pod ctest-nginx-pod-46821351 not present 2026-03-24 16:28:08,970 - INFO - Creating Pod ctest-nginx-pod-46821351 2026-03-24 16:28:09,007 - DEBUG - Pod : ctest-nginx-pod-46821351 UUID is 1de4f6d9-97a2-4bae-a8d3-a8e414cb3467 2026-03-24 16:28:09,010 - DEBUG - Pod ctest-nginx-pod-19216766 not present 2026-03-24 16:28:09,010 - INFO - Creating Pod ctest-nginx-pod-19216766 2026-03-24 16:28:09,040 - DEBUG - Pod : ctest-nginx-pod-19216766 UUID is b18bda50-49c0-457a-b8bf-317415bc5edb 2026-03-24 16:28:09,359 - INFO - Created VN __public__, UUID :c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:09,578 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-24 16:28:09,583 - DEBUG - Requesting: http://10.0.0.50:8082/domain/5fbbcfad-86c0-4203-98e7-ff0b04db96be 2026-03-24 16:28:09,602 - DEBUG - Requesting: http://10.0.0.50:8082/project/425dfa0f-076a-49fd-9d13-e1fbcb09c639 2026-03-24 16:28:09,708 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:09,732 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:09,742 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/63255e2c-5475-4d86-8c05-2eff076d5e95 2026-03-24 16:28:09,765 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/63255e2c-5475-4d86-8c05-2eff076d5e95 2026-03-24 16:28:09,778 - DEBUG - Route Targets: [] 2026-03-24 16:28:09,778 - DEBUG - RT names not yet present for VN __public__ 2026-03-24 16:28:14,779 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-24 16:28:14,784 - DEBUG - Requesting: http://10.0.0.50:8082/domain/5fbbcfad-86c0-4203-98e7-ff0b04db96be 2026-03-24 16:28:14,790 - DEBUG - Requesting: http://10.0.0.50:8082/project/425dfa0f-076a-49fd-9d13-e1fbcb09c639 2026-03-24 16:28:14,823 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:14,832 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:14,839 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/63255e2c-5475-4d86-8c05-2eff076d5e95 2026-03-24 16:28:14,845 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/63255e2c-5475-4d86-8c05-2eff076d5e95 2026-03-24 16:28:14,853 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/faa34fc8-ce96-4d2b-b6e2-fafbbda0e699 2026-03-24 16:28:14,862 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-24 16:28:14,862 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:14,869 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/63255e2c-5475-4d86-8c05-2eff076d5e95 2026-03-24 16:28:14,880 - INFO - Verified VN network id 11 for VN c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:14,881 - INFO - Verifications in API Server for VN __public__ passed 2026-03-24 16:28:14,881 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:14,889 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/63255e2c-5475-4d86-8c05-2eff076d5e95 2026-03-24 16:28:14,894 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/63255e2c-5475-4d86-8c05-2eff076d5e95 2026-03-24 16:28:14,899 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/faa34fc8-ce96-4d2b-b6e2-fafbbda0e699 2026-03-24 16:28:14,904 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-24 16:28:14,909 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14300140895558126783', 'uuid-lslong': '9639758110320211172'}, 'enable': 'true', 'created': '2026-03-24T16:28:09', 'last-modified': '2026-03-24T16:28:09', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.433846'} 2026-03-24 16:28:14,909 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 16:28:14,917 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-24 16:28:14,917 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-03-24 16:28:14,918 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-03-24 16:28:14,918 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-24 16:28:14,922 - DEBUG - Requesting: http://10.0.0.50:8082/domain/5fbbcfad-86c0-4203-98e7-ff0b04db96be 2026-03-24 16:28:14,930 - DEBUG - Requesting: http://10.0.0.50:8082/project/425dfa0f-076a-49fd-9d13-e1fbcb09c639 2026-03-24 16:28:14,971 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:14,979 - DEBUG - =>VN __public__ has no policy to be verified 2026-03-24 16:28:14,979 - DEBUG - Verifying the vn in opserver 2026-03-24 16:28:14,979 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.50 2026-03-24 16:28:14,979 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-03-24 16:28:14,991 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-03-24 16:28:14,991 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-03-24 16:28:14,991 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 16:28:15,001 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 16:28:15,006 - DEBUG - Do not have enough data to verify VN in agent 2026-03-24 16:28:15,006 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 16:28:15,026 - DEBUG - VRF ids for VN __public__: {} 2026-03-24 16:28:15,203 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-03-24 16:28:15,432 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-24 16:28:15,437 - DEBUG - Requesting: http://10.0.0.50:8082/domain/5fbbcfad-86c0-4203-98e7-ff0b04db96be 2026-03-24 16:28:15,444 - DEBUG - Requesting: http://10.0.0.50:8082/project/425dfa0f-076a-49fd-9d13-e1fbcb09c639 2026-03-24 16:28:15,508 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:15,519 - DEBUG - Requesting: http://10.0.0.50:8082/floating-ip-pool/99f9fb2a-34dc-4228-9c9f-7f9c6b38b9e7 2026-03-24 16:28:15,525 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-24 16:28:15,528 - DEBUG - Requesting: http://10.0.0.50:8082/domain/5fbbcfad-86c0-4203-98e7-ff0b04db96be 2026-03-24 16:28:15,534 - DEBUG - Requesting: http://10.0.0.50:8082/project/425dfa0f-076a-49fd-9d13-e1fbcb09c639 2026-03-24 16:28:15,568 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/c6744e61-0dc7-4cbf-85c7-4ced7e53b4e4 2026-03-24 16:28:15,577 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-03-24 16:28:15,577 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-03-24 16:28:15,585 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-24 16:28:15,590 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-03-24 16:28:15,590 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-03-24 16:28:15,593 - DEBUG - Ingress ctest-nginx-ingress-16488585 not present 2026-03-24 16:28:15,594 - INFO - Creating Ingress ctest-nginx-ingress-16488585 2026-03-24 16:28:15,600 - INFO - Created Ingress ctest-nginx-ingress-16488585 2026-03-24 16:28:15,604 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-16488585 2026-03-24 16:28:18,610 - DEBUG - For Ingress ctest-nginx-ingress-16488585, Cluster IP: 10.47.255.248, External IPs ['124.202.118.60'] 2026-03-24 16:28:18,610 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-16488585 2026-03-24 16:28:18,610 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=5f547dff-0c5c-49e4-807e-b1cbcb6da9eb 2026-03-24 16:28:18,621 - INFO - Ingress ctest-nginx-ingress-16488585 with uuid 5f547dff-0c5c-49e4-807e-b1cbcb6da9eb found in kube manager 2026-03-24 16:28:18,621 - INFO - Ingress ctest-nginx-ingress-16488585 verification passed 2026-03-24 16:28:18,624 - DEBUG - Pod ctest-busybox-pod-31034408 not present 2026-03-24 16:28:18,625 - INFO - Creating Pod ctest-busybox-pod-31034408 2026-03-24 16:28:18,665 - DEBUG - Pod : ctest-busybox-pod-31034408 UUID is 95609c99-4678-4a28-ae6e-83c7c4ae2687 2026-03-24 16:28:18,673 - DEBUG - Pod ctest-nginx-pod-46821351 not in running state.Currently in Pending 2026-03-24 16:28:23,680 - INFO - Pod ctest-nginx-pod-46821351 is in running state.Got IP 10.47.255.250 2026-03-24 16:28:23,736 - DEBUG - Pod ctest-nginx-pod-46821351 has vmi 731a7d28-279e-11f1-b37c-0242dc29f896 2026-03-24 16:28:23,736 - INFO - Verified pod ctest-nginx-pod-46821351 in contrail-api 2026-03-24 16:28:23,736 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-24 16:28:23,748 - DEBUG - VMI 731a7d28-279e-11f1-b37c-0242dc29f896 is active in agent 10.0.0.50 2026-03-24 16:28:23,748 - INFO - Verified Pod ctest-nginx-pod-46821351 in agent 10.0.0.50 2026-03-24 16:28:23,751 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=1de4f6d9-97a2-4bae-a8d3-a8e414cb3467 2026-03-24 16:28:23,757 - INFO - Pod ctest-nginx-pod-46821351 with uuid 1de4f6d9-97a2-4bae-a8d3-a8e414cb3467 found in kube manager 2026-03-24 16:28:23,757 - INFO - Pod ctest-nginx-pod-46821351 verification passed 2026-03-24 16:28:23,843 - DEBUG - [Pod ctest-nginx-pod-46821351] Cmd: echo ctest-nginx-pod-46821351 > /usr/share/nginx/html/index.html, Output: 2026-03-24 16:28:23,851 - INFO - Pod ctest-nginx-pod-19216766 is in running state.Got IP 10.47.255.249 2026-03-24 16:28:23,919 - DEBUG - Pod ctest-nginx-pod-19216766 has vmi 73c6344c-279e-11f1-b37c-0242dc29f896 2026-03-24 16:28:23,919 - INFO - Verified pod ctest-nginx-pod-19216766 in contrail-api 2026-03-24 16:28:23,919 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-24 16:28:23,927 - DEBUG - VMI 73c6344c-279e-11f1-b37c-0242dc29f896 is active in agent 10.0.0.50 2026-03-24 16:28:23,927 - INFO - Verified Pod ctest-nginx-pod-19216766 in agent 10.0.0.50 2026-03-24 16:28:23,927 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=b18bda50-49c0-457a-b8bf-317415bc5edb 2026-03-24 16:28:23,931 - INFO - Pod ctest-nginx-pod-19216766 with uuid b18bda50-49c0-457a-b8bf-317415bc5edb found in kube manager 2026-03-24 16:28:23,931 - INFO - Pod ctest-nginx-pod-19216766 verification passed 2026-03-24 16:28:24,024 - DEBUG - [Pod ctest-nginx-pod-19216766] Cmd: echo ctest-nginx-pod-19216766 > /usr/share/nginx/html/index.html, Output: 2026-03-24 16:28:24,031 - DEBUG - Pod ctest-busybox-pod-31034408 not in running state.Currently in Pending 2026-03-24 16:28:29,038 - DEBUG - Pod ctest-busybox-pod-31034408 not in running state.Currently in Pending 2026-03-24 16:28:34,046 - INFO - Pod ctest-busybox-pod-31034408 is in running state.Got IP 10.47.255.247 2026-03-24 16:28:34,098 - DEBUG - Pod ctest-busybox-pod-31034408 has vmi 7903794c-279e-11f1-b37c-0242dc29f896 2026-03-24 16:28:34,098 - INFO - Verified pod ctest-busybox-pod-31034408 in contrail-api 2026-03-24 16:28:34,098 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-24 16:28:34,107 - DEBUG - VMI 7903794c-279e-11f1-b37c-0242dc29f896 is active in agent 10.0.0.50 2026-03-24 16:28:34,107 - INFO - Verified Pod ctest-busybox-pod-31034408 in agent 10.0.0.50 2026-03-24 16:28:34,107 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=95609c99-4678-4a28-ae6e-83c7c4ae2687 2026-03-24 16:28:34,111 - INFO - Pod ctest-busybox-pod-31034408 with uuid 95609c99-4678-4a28-ae6e-83c7c4ae2687 found in kube manager 2026-03-24 16:28:34,111 - INFO - Pod ctest-busybox-pod-31034408 verification passed 2026-03-24 16:28:34,180 - DEBUG - [Pod ctest-busybox-pod-31034408] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-19216766 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-24 16:28:34,180 - DEBUG - [Pod Pod ctest-busybox-pod-31034408] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-24 16:28:34,180 - DEBUG - [Pod Pod ctest-busybox-pod-31034408] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-19216766 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-24 16:28:34,248 - DEBUG - [Pod ctest-busybox-pod-31034408] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-19216766 2026-03-24 16:28:34,248 - DEBUG - [Pod Pod ctest-busybox-pod-31034408] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-24 16:28:34,248 - DEBUG - [Pod Pod ctest-busybox-pod-31034408] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-19216766 2026-03-24 16:28:34,316 - DEBUG - [Pod ctest-busybox-pod-31034408] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-46821351 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-24 16:28:34,317 - DEBUG - [Pod Pod ctest-busybox-pod-31034408] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-24 16:28:34,317 - DEBUG - [Pod Pod ctest-busybox-pod-31034408] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-46821351 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-24 16:28:34,317 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-46821351': 1, 'ctest-nginx-pod-19216766': 2} 2026-03-24 16:28:34,317 - WARNING - Nothing to delete parallely 2026-03-24 16:28:34,317 - INFO - Deleting pod default:ctest-busybox-pod-31034408 2026-03-24 16:28:34,327 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=95609c99-4678-4a28-ae6e-83c7c4ae2687 2026-03-24 16:28:34,335 - WARNING - Pod uuid 95609c99-4678-4a28-ae6e-83c7c4ae2687 is still seen in agent 10.0.0.50 VM list 2026-03-24 16:28:39,336 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=95609c99-4678-4a28-ae6e-83c7c4ae2687 2026-03-24 16:28:39,341 - WARNING - Pod uuid 95609c99-4678-4a28-ae6e-83c7c4ae2687 is still seen in agent 10.0.0.50 VM list 2026-03-24 16:28:44,342 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=95609c99-4678-4a28-ae6e-83c7c4ae2687 2026-03-24 16:28:44,346 - WARNING - Pod uuid 95609c99-4678-4a28-ae6e-83c7c4ae2687 is still seen in agent 10.0.0.50 VM list 2026-03-24 16:28:49,348 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=95609c99-4678-4a28-ae6e-83c7c4ae2687 2026-03-24 16:28:49,352 - WARNING - Pod uuid 95609c99-4678-4a28-ae6e-83c7c4ae2687 is still seen in agent 10.0.0.50 VM list 2026-03-24 16:28:54,353 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=95609c99-4678-4a28-ae6e-83c7c4ae2687 2026-03-24 16:28:54,358 - WARNING - Pod uuid 95609c99-4678-4a28-ae6e-83c7c4ae2687 is still seen in agent 10.0.0.50 VM list 2026-03-24 16:28:59,358 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=95609c99-4678-4a28-ae6e-83c7c4ae2687 2026-03-24 16:28:59,363 - WARNING - Pod uuid 95609c99-4678-4a28-ae6e-83c7c4ae2687 is still seen in agent 10.0.0.50 VM list 2026-03-24 16:29:04,364 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=95609c99-4678-4a28-ae6e-83c7c4ae2687 2026-03-24 16:29:04,373 - WARNING - Pod uuid 95609c99-4678-4a28-ae6e-83c7c4ae2687 is still seen in agent 10.0.0.50 VM list 2026-03-24 16:29:09,373 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=95609c99-4678-4a28-ae6e-83c7c4ae2687 2026-03-24 16:29:09,386 - DEBUG - Pod 95609c99-4678-4a28-ae6e-83c7c4ae2687 is not in agent 10.0.0.50 VM list 2026-03-24 16:29:09,386 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:7903794c-279e-11f1-b37c-0242dc29f896 2026-03-24 16:29:09,391 - DEBUG - VMI 7903794c-279e-11f1-b37c-0242dc29f896 is removed from agent 10.0.0.50 2026-03-24 16:29:09,391 - INFO - Verified that pod ctest-busybox-pod-31034408 is removed in agent 2026-03-24 16:29:09,391 - INFO - Deleting Ingress : ctest-nginx-ingress-16488585 2026-03-24 16:29:09,658 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=5f547dff-0c5c-49e4-807e-b1cbcb6da9eb 2026-03-24 16:29:09,665 - ERROR - Ingress ctest-nginx-ingress-16488585 with uuid 5f547dff-0c5c-49e4-807e-b1cbcb6da9eb still found in kube manager 2026-03-24 16:29:10,665 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=5f547dff-0c5c-49e4-807e-b1cbcb6da9eb 2026-03-24 16:29:10,675 - ERROR - Ingress ctest-nginx-ingress-16488585 with uuid 5f547dff-0c5c-49e4-807e-b1cbcb6da9eb still found in kube manager 2026-03-24 16:29:11,676 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=5f547dff-0c5c-49e4-807e-b1cbcb6da9eb 2026-03-24 16:29:11,682 - INFO - Ingress ctest-nginx-ingress-16488585 with uuid 5f547dff-0c5c-49e4-807e-b1cbcb6da9eb deleted successfully from kube manager 2026-03-24 16:29:11,682 - INFO - Verifications on Ingress %s cleanup passed 2026-03-24 16:29:11,682 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-03-24 16:29:11,790 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-24 16:29:11,805 - WARNING - Control-node ifmap object for FIP pool __fip_pool_public__ , VN __public__ is found! 2026-03-24 16:29:16,806 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-24 16:29:16,811 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-03-24 16:29:16,811 - INFO - Deleting VN __public__ 2026-03-24 16:29:16,888 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/63255e2c-5475-4d86-8c05-2eff076d5e95 2026-03-24 16:29:16,893 - DEBUG - Response Code: 404 2026-03-24 16:29:16,893 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-24 16:29:16,898 - DEBUG - Requesting: http://10.0.0.50:8082/domain/5fbbcfad-86c0-4203-98e7-ff0b04db96be 2026-03-24 16:29:16,906 - DEBUG - Requesting: http://10.0.0.50:8082/project/425dfa0f-076a-49fd-9d13-e1fbcb09c639 2026-03-24 16:29:16,944 - INFO - Validated that VN __public__ is not found in API Server 2026-03-24 16:29:16,944 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-03-24 16:29:16,949 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 16:29:16,954 - DEBUG - VN __public__ is not present in Agent 10.0.0.50 2026-03-24 16:29:16,954 - INFO - Validated that VN __public__ is not in any agent 2026-03-24 16:29:16,954 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 16:29:16,959 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-24 16:29:16,966 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-24 16:29:16,972 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-03-24 16:29:16,973 - INFO - Deleting pod default:ctest-nginx-pod-19216766 2026-03-24 16:29:16,986 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=b18bda50-49c0-457a-b8bf-317415bc5edb 2026-03-24 16:29:16,994 - WARNING - Pod uuid b18bda50-49c0-457a-b8bf-317415bc5edb is still seen in agent 10.0.0.50 VM list 2026-03-24 16:29:21,995 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=b18bda50-49c0-457a-b8bf-317415bc5edb 2026-03-24 16:29:22,000 - DEBUG - Pod b18bda50-49c0-457a-b8bf-317415bc5edb is not in agent 10.0.0.50 VM list 2026-03-24 16:29:22,000 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:73c6344c-279e-11f1-b37c-0242dc29f896 2026-03-24 16:29:22,006 - DEBUG - VMI 73c6344c-279e-11f1-b37c-0242dc29f896 is removed from agent 10.0.0.50 2026-03-24 16:29:22,006 - INFO - Verified that pod ctest-nginx-pod-19216766 is removed in agent 2026-03-24 16:29:22,006 - INFO - Deleting pod default:ctest-nginx-pod-46821351 2026-03-24 16:29:22,027 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1de4f6d9-97a2-4bae-a8d3-a8e414cb3467 2026-03-24 16:29:22,034 - WARNING - Pod uuid 1de4f6d9-97a2-4bae-a8d3-a8e414cb3467 is still seen in agent 10.0.0.50 VM list 2026-03-24 16:29:27,036 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1de4f6d9-97a2-4bae-a8d3-a8e414cb3467 2026-03-24 16:29:27,041 - DEBUG - Pod 1de4f6d9-97a2-4bae-a8d3-a8e414cb3467 is not in agent 10.0.0.50 VM list 2026-03-24 16:29:27,041 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:731a7d28-279e-11f1-b37c-0242dc29f896 2026-03-24 16:29:27,045 - DEBUG - VMI 731a7d28-279e-11f1-b37c-0242dc29f896 is removed from agent 10.0.0.50 2026-03-24 16:29:27,045 - INFO - Verified that pod ctest-nginx-pod-46821351 is removed in agent 2026-03-24 16:29:27,045 - INFO - Deleting service : ctest-nginx-svc-56768026 2026-03-24 16:29:27,567 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-24 16:29:27,572 - DEBUG - No XMPP flaps were noticed during the test