2026-05-25 01:39:04,982 - DEBUG - [10.0.0.22]: Running cmd : hostname 2026-05-25 01:39:05,259 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-412-1 2026-05-25 01:39:05,259 - DEBUG - [10.0.0.22]: Running cmd : hostname -f 2026-05-25 01:39:05,381 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-412-1.cluster.local 2026-05-25 01:39:05,381 - DEBUG - [10.0.0.22]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-05-25 01:39:05,548 - DEBUG - Output : NAMES contrail_test_ExiKOk53R k8s_autoscaler_dns-autoscaler-595558c478-92rd8_kube-system_257c0c75-a0dd-4276-8a58-37038d143b13_0 k8s_coredns_coredns-77f7cc69db-nc6zg_kube-system_6531a432-1efe-415d-9511-112fe78febb3_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-sl6bt_kube-system_2748d729-a4cf-4972-87ef-92f81e1768e8_1 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-h6wk9_kube-system_40d40b08-e08e-4367-820b-cec8a768c290_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-h6wk9_kube-system_40d40b08-e08e-4367-820b-cec8a768c290_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-h6wk9_kube-system_40d40b08-e08e-4367-820b-cec8a768c290_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-4cvdt_kube-system_88c01520-8236-4926-9469-ae85905d4dfd_0 k8s_opensdn-controller-control-named_opensdn-controller-control-4cvdt_kube-system_88c01520-8236-4926-9469-ae85905d4dfd_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-5qp5l_kube-system_b7609ca1-480c-4787-ae3f-2cb10603d0b1_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-h6wk9_kube-system_40d40b08-e08e-4367-820b-cec8a768c290_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-4cvdt_kube-system_88c01520-8236-4926-9469-ae85905d4dfd_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-fqzf6_kube-system_dba0f16c-6b82-4c6c-a244-bf70365e9892_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-nhvtp_kube-system_3199a336-9904-48e2-a090-290cf966b77c_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-nhvtp_kube-system_3199a336-9904-48e2-a090-290cf966b77c_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-8hf6b_kube-system_36cd4986-3057-483f-842b-9eb0ccb21cea_0 k8s_opensdn-analytics-collector_opensdn-analytics-8hf6b_kube-system_36cd4986-3057-483f-842b-9eb0ccb21cea_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-sl6bt_kube-system_2748d729-a4cf-4972-87ef-92f81e1768e8_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-5qp5l_kube-system_b7609ca1-480c-4787-ae3f-2cb10603d0b1_0 k8s_opensdn-controller-config-api_opensdn-controller-config-h6wk9_kube-system_40d40b08-e08e-4367-820b-cec8a768c290_0 k8s_opensdn-controller-control_opensdn-controller-control-4cvdt_kube-system_88c01520-8236-4926-9469-ae85905d4dfd_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-fqzf6_kube-system_dba0f16c-6b82-4c6c-a244-bf70365e9892_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-nhvtp_kube-system_3199a336-9904-48e2-a090-290cf966b77c_0 k8s_opensdn-analytics-api_opensdn-analytics-8hf6b_kube-system_36cd4986-3057-483f-842b-9eb0ccb21cea_0 k8s_kafka_opensdn-analytics-alarm-sl6bt_kube-system_2748d729-a4cf-4972-87ef-92f81e1768e8_0 k8s_redis_redis-wgqq7_kube-system_16605c8b-cf2c-4647-b215-93e297a99b6d_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-fksfc_kube-system_5988bdd9-0f02-4b6a-b380-4ee109f83afd_0 k8s_opensdn-configdb_opensdn-configdb-fksfc_kube-system_5988bdd9-0f02-4b6a-b380-4ee109f83afd_0 k8s_config-zookeeper_config-zookeeper-h6sr2_kube-system_596ecfd3-2b43-43e0-9251-137f2358a7cb_0 k8s_rabbitmq_rabbitmq-hjn9m_kube-system_67921fa8-dd35-4116-a32a-b84dad5618d0_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-fqzf6_kube-system_dba0f16c-6b82-4c6c-a244-bf70365e9892_0 k8s_opensdn-agent-nodemgr_opensdn-agent-q5dpc_kube-system_fb750e1d-32f3-4f7c-9092-f25f4a6185e8_0 k8s_opensdn-vrouter-agent_opensdn-agent-q5dpc_kube-system_fb750e1d-32f3-4f7c-9092-f25f4a6185e8_0 k8s_opensdn-kube-manager_opensdn-kube-manager-c2vh2_kube-system_be2692cf-37e6-476b-af7b-ef9233c454e8_0 k8s_kube-proxy_kube-proxy-cs2fm_kube-system_25e97394-a98f-492a-b47f-c1e49b6a9d89_0 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-412-1_kube-system_40e4d240e834359337c5f4e8ad8785df_1 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-412-1_kube-system_bd62dda2e0c93e53c3cc41b2af83df7b_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-412-1_kube-system_5e50be8ef91e3ca8c7d5388b1b4c19ab_0 2026-05-25 01:39:05,549 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-05-25 01:39:05,674 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.22/24 2026-05-25 01:39:05,674 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}' 2026-05-25 01:39:05,781 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-412-1.cluster.local 2026-05-25 01:39:05,781 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-05-25 01:39:05,905 - DEBUG - Output : 10.0.0.22/24 2026-05-25 01:39:06,284 - INFO - Using existing project ['default-domain', 'k8s-default'](bc6cd06e-a701-4355-b9b3-5b0b24c09183) 2026-05-25 01:39:06,403 - INFO - Using existing project ['default-domain', 'k8s-default'](bc6cd06e-a701-4355-b9b3-5b0b24c09183) 2026-05-25 01:39:06,530 - INFO - ================================================================================ 2026-05-25 01:39:06,531 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-05-25 01:39:06,531 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-05-25 01:39:06,810 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_AgentXmppConnectionStatusReq? 2026-05-25 01:39:06,816 - DEBUG - Nothing to compare xmpp stats {'10.0.0.22': {'10.0.0.22': '2'}} with 2026-05-25 01:39:06,816 - INFO - Initial checks done. Running the testcase now 2026-05-25 01:39:06,816 - INFO - 2026-05-25 01:39:06,821 - INFO - Namespace default already exists 2026-05-25 01:39:06,822 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-c2vh2_kube-system_be2692cf-37e6-476b-af7b-ef9233c454e8_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-05-25 01:39:06,994 - DEBUG - Output : cluster_project={} 2026-05-25 01:39:07,080 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-05-25 01:39:07,080 - DEBUG - Project uuid in Contrail is bc6cd06e-a701-4355-b9b3-5b0b24c09183 2026-05-25 01:39:07,081 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-c2vh2_kube-system_be2692cf-37e6-476b-af7b-ef9233c454e8_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-05-25 01:39:07,241 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-05-25 01:39:07,241 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_NamespaceDatabaseList?namespace_uuid=89d66b4e-699e-42c5-b5b9-18395aed77b4 2026-05-25 01:39:07,246 - INFO - Namespace default with uuid 89d66b4e-699e-42c5-b5b9-18395aed77b4 found in kube manager 2026-05-25 01:39:07,246 - INFO - Namespace default verification passed 2026-05-25 01:39:07,250 - DEBUG - Service ctest-nginx-svc-42564610 not present 2026-05-25 01:39:07,250 - INFO - Creating service ctest-nginx-svc-42564610 2026-05-25 01:39:07,264 - INFO - Created Service ctest-nginx-svc-42564610 2026-05-25 01:39:07,268 - DEBUG - Pod ctest-nginx-pod-01464846 not present 2026-05-25 01:39:07,269 - INFO - Creating Pod ctest-nginx-pod-01464846 2026-05-25 01:39:07,304 - DEBUG - Pod : ctest-nginx-pod-01464846 UUID is c2e16cb0-2701-4bf4-ac61-03d49257c441 2026-05-25 01:39:07,308 - DEBUG - Pod ctest-nginx-pod-57651952 not present 2026-05-25 01:39:07,308 - INFO - Creating Pod ctest-nginx-pod-57651952 2026-05-25 01:39:07,338 - DEBUG - Pod : ctest-nginx-pod-57651952 UUID is be365c4d-2cb3-4111-82be-092cadd4a63c 2026-05-25 01:39:07,676 - INFO - Created VN __public__, UUID :35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:07,904 - DEBUG - Requesting: http://10.0.0.22:8082/domains 2026-05-25 01:39:07,913 - DEBUG - Requesting: http://10.0.0.22:8082/domain/0adcf986-fde3-4f3d-8710-acec42636888 2026-05-25 01:39:07,995 - DEBUG - Requesting: http://10.0.0.22:8082/project/bc6cd06e-a701-4355-b9b3-5b0b24c09183 2026-05-25 01:39:08,083 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:08,094 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:08,102 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/9913b7ed-dd7b-43ff-a3e2-5d1394a7f4ae 2026-05-25 01:39:08,110 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/9913b7ed-dd7b-43ff-a3e2-5d1394a7f4ae 2026-05-25 01:39:08,118 - DEBUG - Route Targets: [] 2026-05-25 01:39:08,118 - DEBUG - RT names not yet present for VN __public__ 2026-05-25 01:39:13,119 - DEBUG - Requesting: http://10.0.0.22:8082/domains 2026-05-25 01:39:13,130 - DEBUG - Requesting: http://10.0.0.22:8082/domain/0adcf986-fde3-4f3d-8710-acec42636888 2026-05-25 01:39:13,140 - DEBUG - Requesting: http://10.0.0.22:8082/project/bc6cd06e-a701-4355-b9b3-5b0b24c09183 2026-05-25 01:39:13,211 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:13,223 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:13,231 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/9913b7ed-dd7b-43ff-a3e2-5d1394a7f4ae 2026-05-25 01:39:13,239 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/9913b7ed-dd7b-43ff-a3e2-5d1394a7f4ae 2026-05-25 01:39:13,245 - DEBUG - Requesting: http://10.0.0.22:8082/route-target/371f045a-40d0-4ef3-8fd3-00795ca384de 2026-05-25 01:39:13,251 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-05-25 01:39:13,251 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:13,259 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/9913b7ed-dd7b-43ff-a3e2-5d1394a7f4ae 2026-05-25 01:39:13,270 - INFO - Verified VN network id 11 for VN 35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:13,271 - INFO - Verifications in API Server for VN __public__ passed 2026-05-25 01:39:13,271 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:13,278 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/9913b7ed-dd7b-43ff-a3e2-5d1394a7f4ae 2026-05-25 01:39:13,283 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/9913b7ed-dd7b-43ff-a3e2-5d1394a7f4ae 2026-05-25 01:39:13,289 - DEBUG - Requesting: http://10.0.0.22:8082/route-target/371f045a-40d0-4ef3-8fd3-00795ca384de 2026-05-25 01:39:13,296 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-05-25 01:39:13,305 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3852121877734640871', 'uuid-lslong': '12764015522422315103'}, 'enable': 'true', 'created': '2026-05-25T01:39:07', 'last-modified': '2026-05-25T01:39:07', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.509051'} 2026-05-25 01:39:13,305 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-25 01:39:13,315 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-05-25 01:39:13,315 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-05-25 01:39:13,317 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-05-25 01:39:13,318 - DEBUG - Requesting: http://10.0.0.22:8082/domains 2026-05-25 01:39:13,323 - DEBUG - Requesting: http://10.0.0.22:8082/domain/0adcf986-fde3-4f3d-8710-acec42636888 2026-05-25 01:39:13,329 - DEBUG - Requesting: http://10.0.0.22:8082/project/bc6cd06e-a701-4355-b9b3-5b0b24c09183 2026-05-25 01:39:13,357 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:13,365 - DEBUG - =>VN __public__ has no policy to be verified 2026-05-25 01:39:13,365 - DEBUG - Verifying the vn in opserver 2026-05-25 01:39:13,365 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.22 2026-05-25 01:39:13,365 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks 2026-05-25 01:39:13,369 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-05-25 01:39:13,369 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-05-25 01:39:13,370 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-25 01:39:13,375 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-25 01:39:13,379 - DEBUG - Do not have enough data to verify VN in agent 2026-05-25 01:39:13,379 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-25 01:39:13,383 - DEBUG - VRF ids for VN __public__: {} 2026-05-25 01:39:13,544 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-05-25 01:39:13,751 - DEBUG - Requesting: http://10.0.0.22:8082/domains 2026-05-25 01:39:13,761 - DEBUG - Requesting: http://10.0.0.22:8082/domain/0adcf986-fde3-4f3d-8710-acec42636888 2026-05-25 01:39:13,776 - DEBUG - Requesting: http://10.0.0.22:8082/project/bc6cd06e-a701-4355-b9b3-5b0b24c09183 2026-05-25 01:39:13,845 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:13,864 - DEBUG - Requesting: http://10.0.0.22:8082/floating-ip-pool/e2006a8a-b927-4d79-9078-927b45b702cb 2026-05-25 01:39:13,872 - DEBUG - Requesting: http://10.0.0.22:8082/domains 2026-05-25 01:39:13,891 - DEBUG - Requesting: http://10.0.0.22:8082/domain/0adcf986-fde3-4f3d-8710-acec42636888 2026-05-25 01:39:13,899 - DEBUG - Requesting: http://10.0.0.22:8082/project/bc6cd06e-a701-4355-b9b3-5b0b24c09183 2026-05-25 01:39:13,943 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/35757c70-39e2-4ce7-b122-e47db18dd85f 2026-05-25 01:39:13,955 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-05-25 01:39:13,955 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-05-25 01:39:13,963 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-05-25 01:39:13,971 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-05-25 01:39:13,971 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-05-25 01:39:13,975 - DEBUG - Ingress ctest-nginx-ingress-98150334 not present 2026-05-25 01:39:13,976 - INFO - Creating Ingress ctest-nginx-ingress-98150334 2026-05-25 01:39:13,982 - INFO - Created Ingress ctest-nginx-ingress-98150334 2026-05-25 01:39:13,986 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-98150334 2026-05-25 01:39:16,992 - DEBUG - For Ingress ctest-nginx-ingress-98150334, Cluster IP: 10.47.255.248, External IPs ['191.31.191.188'] 2026-05-25 01:39:16,992 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-98150334 2026-05-25 01:39:16,992 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=39455d43-7aac-4c20-8832-3cad407b9481 2026-05-25 01:39:16,997 - INFO - Ingress ctest-nginx-ingress-98150334 with uuid 39455d43-7aac-4c20-8832-3cad407b9481 found in kube manager 2026-05-25 01:39:16,997 - INFO - Ingress ctest-nginx-ingress-98150334 verification passed 2026-05-25 01:39:17,001 - DEBUG - Pod ctest-busybox-pod-68798362 not present 2026-05-25 01:39:17,001 - INFO - Creating Pod ctest-busybox-pod-68798362 2026-05-25 01:39:17,023 - DEBUG - Node for Pod ctest-busybox-pod-68798362 not yet populated 2026-05-25 01:39:20,036 - DEBUG - Pod : ctest-busybox-pod-68798362 UUID is 74c6a81c-8fe4-41d7-9516-7b035c3adc33 2026-05-25 01:39:20,043 - DEBUG - Pod ctest-nginx-pod-01464846 not in running state.Currently in Pending 2026-05-25 01:39:25,051 - INFO - Pod ctest-nginx-pod-01464846 is in running state.Got IP 10.47.255.250 2026-05-25 01:39:25,103 - DEBUG - Pod ctest-nginx-pod-01464846 has vmi 86d00916-57da-11f1-b311-024282ce14d1 2026-05-25 01:39:25,104 - INFO - Verified pod ctest-nginx-pod-01464846 in contrail-api 2026-05-25 01:39:25,104 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-05-25 01:39:25,114 - DEBUG - VMI 86d00916-57da-11f1-b311-024282ce14d1 is active in agent 10.0.0.22 2026-05-25 01:39:25,114 - INFO - Verified Pod ctest-nginx-pod-01464846 in agent 10.0.0.22 2026-05-25 01:39:25,114 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=c2e16cb0-2701-4bf4-ac61-03d49257c441 2026-05-25 01:39:25,118 - INFO - Pod ctest-nginx-pod-01464846 with uuid c2e16cb0-2701-4bf4-ac61-03d49257c441 found in kube manager 2026-05-25 01:39:25,118 - INFO - Pod ctest-nginx-pod-01464846 verification passed 2026-05-25 01:39:25,199 - DEBUG - [Pod ctest-nginx-pod-01464846] Cmd: echo ctest-nginx-pod-01464846 > /usr/share/nginx/html/index.html, Output: 2026-05-25 01:39:25,207 - INFO - Pod ctest-nginx-pod-57651952 is in running state.Got IP 10.47.255.249 2026-05-25 01:39:25,259 - DEBUG - Pod ctest-nginx-pod-57651952 has vmi 878b29da-57da-11f1-b311-024282ce14d1 2026-05-25 01:39:25,260 - INFO - Verified pod ctest-nginx-pod-57651952 in contrail-api 2026-05-25 01:39:25,260 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-05-25 01:39:25,268 - DEBUG - VMI 878b29da-57da-11f1-b311-024282ce14d1 is active in agent 10.0.0.22 2026-05-25 01:39:25,268 - INFO - Verified Pod ctest-nginx-pod-57651952 in agent 10.0.0.22 2026-05-25 01:39:25,268 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=be365c4d-2cb3-4111-82be-092cadd4a63c 2026-05-25 01:39:25,272 - INFO - Pod ctest-nginx-pod-57651952 with uuid be365c4d-2cb3-4111-82be-092cadd4a63c found in kube manager 2026-05-25 01:39:25,272 - INFO - Pod ctest-nginx-pod-57651952 verification passed 2026-05-25 01:39:25,354 - DEBUG - [Pod ctest-nginx-pod-57651952] Cmd: echo ctest-nginx-pod-57651952 > /usr/share/nginx/html/index.html, Output: 2026-05-25 01:39:25,363 - DEBUG - Pod ctest-busybox-pod-68798362 not in running state.Currently in Pending 2026-05-25 01:39:30,373 - INFO - Pod ctest-busybox-pod-68798362 is in running state.Got IP 10.47.255.247 2026-05-25 01:39:30,427 - DEBUG - Pod ctest-busybox-pod-68798362 has vmi 8c7548b8-57da-11f1-b311-024282ce14d1 2026-05-25 01:39:30,428 - INFO - Verified pod ctest-busybox-pod-68798362 in contrail-api 2026-05-25 01:39:30,428 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-05-25 01:39:30,437 - DEBUG - VMI 8c7548b8-57da-11f1-b311-024282ce14d1 is active in agent 10.0.0.22 2026-05-25 01:39:30,437 - INFO - Verified Pod ctest-busybox-pod-68798362 in agent 10.0.0.22 2026-05-25 01:39:30,437 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=74c6a81c-8fe4-41d7-9516-7b035c3adc33 2026-05-25 01:39:30,442 - INFO - Pod ctest-busybox-pod-68798362 with uuid 74c6a81c-8fe4-41d7-9516-7b035c3adc33 found in kube manager 2026-05-25 01:39:30,442 - INFO - Pod ctest-busybox-pod-68798362 verification passed 2026-05-25 01:39:30,532 - DEBUG - [Pod ctest-busybox-pod-68798362] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-01464846 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-25 01:39:30,532 - DEBUG - [Pod Pod ctest-busybox-pod-68798362] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-05-25 01:39:30,532 - DEBUG - [Pod Pod ctest-busybox-pod-68798362] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-01464846 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-25 01:39:30,604 - DEBUG - [Pod ctest-busybox-pod-68798362] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-01464846 2026-05-25 01:39:30,604 - DEBUG - [Pod Pod ctest-busybox-pod-68798362] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-05-25 01:39:30,604 - DEBUG - [Pod Pod ctest-busybox-pod-68798362] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-01464846 2026-05-25 01:39:30,671 - DEBUG - [Pod ctest-busybox-pod-68798362] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-01464846 2026-05-25 01:39:30,671 - DEBUG - [Pod Pod ctest-busybox-pod-68798362] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-05-25 01:39:30,671 - DEBUG - [Pod Pod ctest-busybox-pod-68798362] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-01464846 2026-05-25 01:39:30,741 - DEBUG - [Pod ctest-busybox-pod-68798362] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-01464846 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-25 01:39:30,741 - DEBUG - [Pod Pod ctest-busybox-pod-68798362] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-05-25 01:39:30,741 - DEBUG - [Pod Pod ctest-busybox-pod-68798362] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-01464846 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-25 01:39:30,822 - DEBUG - [Pod ctest-busybox-pod-68798362] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-57651952 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-25 01:39:30,822 - DEBUG - [Pod Pod ctest-busybox-pod-68798362] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-05-25 01:39:30,822 - DEBUG - [Pod Pod ctest-busybox-pod-68798362] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-57651952 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-25 01:39:30,822 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-01464846': 4, 'ctest-nginx-pod-57651952': 1} 2026-05-25 01:39:30,822 - WARNING - Nothing to delete parallely 2026-05-25 01:39:30,822 - INFO - Deleting pod default:ctest-busybox-pod-68798362 2026-05-25 01:39:30,840 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=74c6a81c-8fe4-41d7-9516-7b035c3adc33 2026-05-25 01:39:30,845 - WARNING - Pod uuid 74c6a81c-8fe4-41d7-9516-7b035c3adc33 is still seen in agent 10.0.0.22 VM list 2026-05-25 01:39:35,846 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=74c6a81c-8fe4-41d7-9516-7b035c3adc33 2026-05-25 01:39:35,851 - WARNING - Pod uuid 74c6a81c-8fe4-41d7-9516-7b035c3adc33 is still seen in agent 10.0.0.22 VM list 2026-05-25 01:39:40,852 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=74c6a81c-8fe4-41d7-9516-7b035c3adc33 2026-05-25 01:39:40,858 - WARNING - Pod uuid 74c6a81c-8fe4-41d7-9516-7b035c3adc33 is still seen in agent 10.0.0.22 VM list 2026-05-25 01:39:45,859 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=74c6a81c-8fe4-41d7-9516-7b035c3adc33 2026-05-25 01:39:45,865 - WARNING - Pod uuid 74c6a81c-8fe4-41d7-9516-7b035c3adc33 is still seen in agent 10.0.0.22 VM list 2026-05-25 01:39:50,865 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=74c6a81c-8fe4-41d7-9516-7b035c3adc33 2026-05-25 01:39:50,871 - WARNING - Pod uuid 74c6a81c-8fe4-41d7-9516-7b035c3adc33 is still seen in agent 10.0.0.22 VM list 2026-05-25 01:39:55,871 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=74c6a81c-8fe4-41d7-9516-7b035c3adc33 2026-05-25 01:39:55,878 - WARNING - Pod uuid 74c6a81c-8fe4-41d7-9516-7b035c3adc33 is still seen in agent 10.0.0.22 VM list 2026-05-25 01:40:00,879 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=74c6a81c-8fe4-41d7-9516-7b035c3adc33 2026-05-25 01:40:00,885 - WARNING - Pod uuid 74c6a81c-8fe4-41d7-9516-7b035c3adc33 is still seen in agent 10.0.0.22 VM list 2026-05-25 01:40:05,887 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=74c6a81c-8fe4-41d7-9516-7b035c3adc33 2026-05-25 01:40:05,892 - DEBUG - Pod 74c6a81c-8fe4-41d7-9516-7b035c3adc33 is not in agent 10.0.0.22 VM list 2026-05-25 01:40:05,892 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:8c7548b8-57da-11f1-b311-024282ce14d1 2026-05-25 01:40:05,896 - DEBUG - VMI 8c7548b8-57da-11f1-b311-024282ce14d1 is removed from agent 10.0.0.22 2026-05-25 01:40:05,896 - INFO - Verified that pod ctest-busybox-pod-68798362 is removed in agent 2026-05-25 01:40:05,896 - INFO - Deleting Ingress : ctest-nginx-ingress-98150334 2026-05-25 01:40:05,903 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=39455d43-7aac-4c20-8832-3cad407b9481 2026-05-25 01:40:05,909 - ERROR - Ingress ctest-nginx-ingress-98150334 with uuid 39455d43-7aac-4c20-8832-3cad407b9481 still found in kube manager 2026-05-25 01:40:06,909 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=39455d43-7aac-4c20-8832-3cad407b9481 2026-05-25 01:40:06,916 - ERROR - Ingress ctest-nginx-ingress-98150334 with uuid 39455d43-7aac-4c20-8832-3cad407b9481 still found in kube manager 2026-05-25 01:40:07,917 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=39455d43-7aac-4c20-8832-3cad407b9481 2026-05-25 01:40:07,923 - INFO - Ingress ctest-nginx-ingress-98150334 with uuid 39455d43-7aac-4c20-8832-3cad407b9481 deleted successfully from kube manager 2026-05-25 01:40:07,923 - INFO - Verifications on Ingress %s cleanup passed 2026-05-25 01:40:07,923 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-05-25 01:40:08,043 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-05-25 01:40:08,059 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-05-25 01:40:08,060 - INFO - Deleting VN __public__ 2026-05-25 01:40:08,204 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/9913b7ed-dd7b-43ff-a3e2-5d1394a7f4ae 2026-05-25 01:40:08,210 - DEBUG - Response Code: 404 2026-05-25 01:40:08,210 - DEBUG - Requesting: http://10.0.0.22:8082/domains 2026-05-25 01:40:08,214 - DEBUG - Requesting: http://10.0.0.22:8082/domain/0adcf986-fde3-4f3d-8710-acec42636888 2026-05-25 01:40:08,224 - DEBUG - Requesting: http://10.0.0.22:8082/project/bc6cd06e-a701-4355-b9b3-5b0b24c09183 2026-05-25 01:40:08,283 - INFO - Validated that VN __public__ is not found in API Server 2026-05-25 01:40:08,283 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-05-25 01:40:08,288 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-25 01:40:08,293 - DEBUG - VN __public__ is not present in Agent 10.0.0.22 2026-05-25 01:40:08,293 - INFO - Validated that VN __public__ is not in any agent 2026-05-25 01:40:08,293 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-25 01:40:08,298 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-25 01:40:08,319 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-05-25 01:40:08,326 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-05-25 01:40:08,326 - INFO - Deleting pod default:ctest-nginx-pod-57651952 2026-05-25 01:40:08,340 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=be365c4d-2cb3-4111-82be-092cadd4a63c 2026-05-25 01:40:08,347 - WARNING - Pod uuid be365c4d-2cb3-4111-82be-092cadd4a63c is still seen in agent 10.0.0.22 VM list 2026-05-25 01:40:13,348 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=be365c4d-2cb3-4111-82be-092cadd4a63c 2026-05-25 01:40:13,355 - DEBUG - Pod be365c4d-2cb3-4111-82be-092cadd4a63c is not in agent 10.0.0.22 VM list 2026-05-25 01:40:13,355 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:878b29da-57da-11f1-b311-024282ce14d1 2026-05-25 01:40:13,361 - DEBUG - VMI 878b29da-57da-11f1-b311-024282ce14d1 is removed from agent 10.0.0.22 2026-05-25 01:40:13,361 - INFO - Verified that pod ctest-nginx-pod-57651952 is removed in agent 2026-05-25 01:40:13,361 - INFO - Deleting pod default:ctest-nginx-pod-01464846 2026-05-25 01:40:13,388 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=c2e16cb0-2701-4bf4-ac61-03d49257c441 2026-05-25 01:40:13,396 - WARNING - Pod uuid c2e16cb0-2701-4bf4-ac61-03d49257c441 is still seen in agent 10.0.0.22 VM list 2026-05-25 01:40:18,397 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_VmListReq?uuid=c2e16cb0-2701-4bf4-ac61-03d49257c441 2026-05-25 01:40:18,402 - DEBUG - Pod c2e16cb0-2701-4bf4-ac61-03d49257c441 is not in agent 10.0.0.22 VM list 2026-05-25 01:40:18,402 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:86d00916-57da-11f1-b311-024282ce14d1 2026-05-25 01:40:18,407 - DEBUG - VMI 86d00916-57da-11f1-b311-024282ce14d1 is removed from agent 10.0.0.22 2026-05-25 01:40:18,407 - INFO - Verified that pod ctest-nginx-pod-01464846 is removed in agent 2026-05-25 01:40:18,407 - INFO - Deleting service : ctest-nginx-svc-42564610 2026-05-25 01:40:18,930 - DEBUG - Requesting: http://10.0.0.22:8085/Snh_AgentXmppConnectionStatusReq? 2026-05-25 01:40:18,935 - DEBUG - No XMPP flaps were noticed during the test