2026-02-05 01:39:33,192 - DEBUG - [10.0.0.33]: Running cmd : hostname 2026-02-05 01:39:33,447 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-294-1 2026-02-05 01:39:33,447 - DEBUG - [10.0.0.33]: Running cmd : hostname -f 2026-02-05 01:39:33,576 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local 2026-02-05 01:39:33,577 - DEBUG - [10.0.0.33]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-05 01:39:33,749 - DEBUG - Output : NAMES contrail_test_xGR4gA1cz k8s_autoscaler_dns-autoscaler-595558c478-lskj8_kube-system_ddd71b1f-2038-4fbe-960c-6c6871dc694c_0 k8s_coredns_coredns-77f7cc69db-dq4dp_kube-system_960fcb9f-ee23-4f4e-9202-fffb513b1949_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-cjpwr_kube-system_8eb6b9b0-7026-467f-98b0-85502ca3c6d9_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-cjpwr_kube-system_8eb6b9b0-7026-467f-98b0-85502ca3c6d9_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-dgw4f_kube-system_3f471163-1001-4be1-bb98-6bd49e664b37_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-pk86m_kube-system_ef1b20c0-089c-4a08-bcb9-33d8e016457a_0 k8s_opensdn-controller-control-named_opensdn-controller-control-pk86m_kube-system_ef1b20c0-089c-4a08-bcb9-33d8e016457a_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-cjpwr_kube-system_8eb6b9b0-7026-467f-98b0-85502ca3c6d9_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-dgw4f_kube-system_3f471163-1001-4be1-bb98-6bd49e664b37_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-pk86m_kube-system_ef1b20c0-089c-4a08-bcb9-33d8e016457a_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-skk9m_kube-system_ee096dca-2962-4444-a17c-98c763d07928_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-skk9m_kube-system_ee096dca-2962-4444-a17c-98c763d07928_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-9dnw4_kube-system_97eae968-4367-47e9-9b6a-a261690627c0_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-9dnw4_kube-system_97eae968-4367-47e9-9b6a-a261690627c0_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-cjpwr_kube-system_8eb6b9b0-7026-467f-98b0-85502ca3c6d9_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-fjjz4_kube-system_1bee55df-754f-46a7-a8bc-01de83bf00f3_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-6bnm8_kube-system_1b25aeb0-7a17-438a-a21b-ad20db630b47_0 k8s_opensdn-analytics-collector_opensdn-analytics-6bnm8_kube-system_1b25aeb0-7a17-438a-a21b-ad20db630b47_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-kcc4f_kube-system_6653372f-9d58-46d9-96bc-2278f0e4dd85_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-dgw4f_kube-system_3f471163-1001-4be1-bb98-6bd49e664b37_0 k8s_opensdn-controller-control_opensdn-controller-control-pk86m_kube-system_ef1b20c0-089c-4a08-bcb9-33d8e016457a_0 k8s_kafka_opensdn-analytics-alarm-skk9m_kube-system_ee096dca-2962-4444-a17c-98c763d07928_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-9dnw4_kube-system_97eae968-4367-47e9-9b6a-a261690627c0_0 k8s_opensdn-controller-config-api_opensdn-controller-config-cjpwr_kube-system_8eb6b9b0-7026-467f-98b0-85502ca3c6d9_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-fjjz4_kube-system_1bee55df-754f-46a7-a8bc-01de83bf00f3_0 k8s_opensdn-analytics-api_opensdn-analytics-6bnm8_kube-system_1b25aeb0-7a17-438a-a21b-ad20db630b47_0 k8s_redis_redis-q9h4z_kube-system_8cbf7f97-0d71-4e56-9758-1ad81175e2a1_0 k8s_rabbitmq_rabbitmq-czztj_kube-system_fa0413de-8548-4f3d-ab71-17117dc24844_0 k8s_opensdn-agent-nodemgr_opensdn-agent-l44k5_kube-system_5272f395-5fd4-49d5-a1f3-65fa842dc77b_0 k8s_config-zookeeper_config-zookeeper-zsq5f_kube-system_b2835475-d95a-422e-ad99-b6fa7b5d12d5_0 k8s_opensdn-configdb_opensdn-configdb-kcc4f_kube-system_6653372f-9d58-46d9-96bc-2278f0e4dd85_0 k8s_opensdn-vrouter-agent_opensdn-agent-l44k5_kube-system_5272f395-5fd4-49d5-a1f3-65fa842dc77b_0 k8s_opensdn-kube-manager_opensdn-kube-manager-5c7bd_kube-system_ba4dbbcf-9b4c-474e-a0cd-4abe21d07dd8_0 k8s_kube-proxy_kube-proxy-cgr6f_kube-system_6f8d0e93-f9cb-46da-bc63-129ab9015c08_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-294-1_kube-system_c834026160e58ba4a1c2f77cf6004268_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-294-1_kube-system_4eb5bace47fb14ca1b83e40c5dba182c_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-294-1_kube-system_2ffa052fa0f28535419fa1d08458f576_0 2026-02-05 01:39:33,750 - DEBUG - [10.0.0.33]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-02-05 01:39:33,881 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.33/24 2026-02-05 01:39:33,881 - DEBUG - [10.0.0.33]: Running cmd : getent hosts 10.0.0.33 | head -n 1 | awk '{print $2}' 2026-02-05 01:39:34,018 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-294-1.cluster.local 2026-02-05 01:39:34,018 - DEBUG - [10.0.0.33]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-02-05 01:39:34,140 - DEBUG - Output : 10.0.0.33/24 2026-02-05 01:39:34,519 - INFO - Using existing project ['default-domain', 'k8s-default'](b0f57af3-d71e-404d-9ab8-ef54213610e1) 2026-02-05 01:39:34,668 - INFO - Using existing project ['default-domain', 'k8s-default'](b0f57af3-d71e-404d-9ab8-ef54213610e1) 2026-02-05 01:39:34,813 - INFO - ================================================================================ 2026-02-05 01:39:34,813 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-02-05 01:39:34,813 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-02-05 01:39:35,091 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-05 01:39:35,098 - DEBUG - Nothing to compare xmpp stats {'10.0.0.33': {'10.0.0.33': '2'}} with 2026-02-05 01:39:35,098 - INFO - Initial checks done. Running the testcase now 2026-02-05 01:39:35,098 - INFO - 2026-02-05 01:39:35,104 - INFO - Namespace default already exists 2026-02-05 01:39:35,104 - DEBUG - [10.0.0.33]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-5c7bd_kube-system_ba4dbbcf-9b4c-474e-a0cd-4abe21d07dd8_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-05 01:39:35,274 - DEBUG - Output : cluster_project={} 2026-02-05 01:39:35,365 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-02-05 01:39:35,365 - DEBUG - Project uuid in Contrail is b0f57af3-d71e-404d-9ab8-ef54213610e1 2026-02-05 01:39:35,365 - DEBUG - [10.0.0.33]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-5c7bd_kube-system_ba4dbbcf-9b4c-474e-a0cd-4abe21d07dd8_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-02-05 01:39:35,555 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-02-05 01:39:35,555 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_NamespaceDatabaseList?namespace_uuid=1208ae77-d3c7-427d-9cc5-536e6c814731 2026-02-05 01:39:35,559 - INFO - Namespace default with uuid 1208ae77-d3c7-427d-9cc5-536e6c814731 found in kube manager 2026-02-05 01:39:35,559 - INFO - Namespace default verification passed 2026-02-05 01:39:35,562 - DEBUG - Service ctest-nginx-svc-44055259 not present 2026-02-05 01:39:35,563 - INFO - Creating service ctest-nginx-svc-44055259 2026-02-05 01:39:35,578 - INFO - Created Service ctest-nginx-svc-44055259 2026-02-05 01:39:35,582 - DEBUG - Pod ctest-nginx-pod-38910120 not present 2026-02-05 01:39:35,583 - INFO - Creating Pod ctest-nginx-pod-38910120 2026-02-05 01:39:35,610 - DEBUG - Pod : ctest-nginx-pod-38910120 UUID is cba5bdf6-98be-4696-91a5-825d0e98e56b 2026-02-05 01:39:35,613 - DEBUG - Pod ctest-nginx-pod-55172250 not present 2026-02-05 01:39:35,614 - INFO - Creating Pod ctest-nginx-pod-55172250 2026-02-05 01:39:35,640 - DEBUG - Pod : ctest-nginx-pod-55172250 UUID is 67c9264b-7a7a-4aad-ac90-51dfc3f570cc 2026-02-05 01:39:35,942 - INFO - Created VN __public__, UUID :797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:36,200 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-05 01:39:36,206 - DEBUG - Requesting: http://10.0.0.33:8082/domain/2738dc43-b459-4bf5-951e-64e8b58fb46c 2026-02-05 01:39:36,278 - DEBUG - Requesting: http://10.0.0.33:8082/project/b0f57af3-d71e-404d-9ab8-ef54213610e1 2026-02-05 01:39:36,385 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:36,397 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:36,406 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/d1728205-7240-4f90-b9c8-857437b00f5f 2026-02-05 01:39:36,413 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/d1728205-7240-4f90-b9c8-857437b00f5f 2026-02-05 01:39:36,420 - DEBUG - Route Targets: [] 2026-02-05 01:39:36,420 - DEBUG - RT names not yet present for VN __public__ 2026-02-05 01:39:41,421 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-05 01:39:41,426 - DEBUG - Requesting: http://10.0.0.33:8082/domain/2738dc43-b459-4bf5-951e-64e8b58fb46c 2026-02-05 01:39:41,433 - DEBUG - Requesting: http://10.0.0.33:8082/project/b0f57af3-d71e-404d-9ab8-ef54213610e1 2026-02-05 01:39:41,463 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:41,473 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:41,482 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/d1728205-7240-4f90-b9c8-857437b00f5f 2026-02-05 01:39:41,490 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/d1728205-7240-4f90-b9c8-857437b00f5f 2026-02-05 01:39:41,495 - DEBUG - Requesting: http://10.0.0.33:8082/route-target/fc68eb1d-8546-4575-bc3d-ed260e2ccefa 2026-02-05 01:39:41,501 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-05 01:39:41,501 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:41,509 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/d1728205-7240-4f90-b9c8-857437b00f5f 2026-02-05 01:39:41,523 - INFO - Verified VN network id 11 for VN 797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:41,523 - INFO - Verifications in API Server for VN __public__ passed 2026-02-05 01:39:41,523 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:41,533 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/d1728205-7240-4f90-b9c8-857437b00f5f 2026-02-05 01:39:41,541 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/d1728205-7240-4f90-b9c8-857437b00f5f 2026-02-05 01:39:41,548 - DEBUG - Requesting: http://10.0.0.33:8082/route-target/fc68eb1d-8546-4575-bc3d-ed260e2ccefa 2026-02-05 01:39:41,554 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-05 01:39:41,559 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8754066677165475918', 'uuid-lslong': '12954433037125880813'}, 'enable': 'true', 'created': '2026-02-05T01:39:35', 'last-modified': '2026-02-05T01:39:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.448147'} 2026-02-05 01:39:41,560 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-05 01:39:41,569 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-05 01:39:41,569 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-02-05 01:39:41,569 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-02-05 01:39:41,570 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-05 01:39:41,575 - DEBUG - Requesting: http://10.0.0.33:8082/domain/2738dc43-b459-4bf5-951e-64e8b58fb46c 2026-02-05 01:39:41,582 - DEBUG - Requesting: http://10.0.0.33:8082/project/b0f57af3-d71e-404d-9ab8-ef54213610e1 2026-02-05 01:39:41,612 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:41,620 - DEBUG - =>VN __public__ has no policy to be verified 2026-02-05 01:39:41,620 - DEBUG - Verifying the vn in opserver 2026-02-05 01:39:41,620 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.33 2026-02-05 01:39:41,620 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-02-05 01:39:41,625 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-02-05 01:39:41,625 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-02-05 01:39:41,625 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-05 01:39:41,631 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-05 01:39:41,635 - DEBUG - Do not have enough data to verify VN in agent 2026-02-05 01:39:41,635 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-05 01:39:41,640 - DEBUG - VRF ids for VN __public__: {} 2026-02-05 01:39:41,801 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-02-05 01:39:42,007 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-05 01:39:42,013 - DEBUG - Requesting: http://10.0.0.33:8082/domain/2738dc43-b459-4bf5-951e-64e8b58fb46c 2026-02-05 01:39:42,019 - DEBUG - Requesting: http://10.0.0.33:8082/project/b0f57af3-d71e-404d-9ab8-ef54213610e1 2026-02-05 01:39:42,062 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:42,073 - DEBUG - Requesting: http://10.0.0.33:8082/floating-ip-pool/a5d6c701-0a61-4b85-85a0-fc6b0ce9a7a5 2026-02-05 01:39:42,079 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-05 01:39:42,084 - DEBUG - Requesting: http://10.0.0.33:8082/domain/2738dc43-b459-4bf5-951e-64e8b58fb46c 2026-02-05 01:39:42,090 - DEBUG - Requesting: http://10.0.0.33:8082/project/b0f57af3-d71e-404d-9ab8-ef54213610e1 2026-02-05 01:39:42,191 - DEBUG - Requesting: http://10.0.0.33:8082/virtual-network/797cb143-0888-484e-b3c7-643634790bed 2026-02-05 01:39:42,203 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-02-05 01:39:42,203 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-02-05 01:39:42,214 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-05 01:39:42,221 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-02-05 01:39:42,221 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-02-05 01:39:42,225 - DEBUG - Ingress ctest-nginx-ingress-36721569 not present 2026-02-05 01:39:42,226 - INFO - Creating Ingress ctest-nginx-ingress-36721569 2026-02-05 01:39:42,235 - INFO - Created Ingress ctest-nginx-ingress-36721569 2026-02-05 01:39:42,242 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-36721569 2026-02-05 01:39:45,251 - DEBUG - For Ingress ctest-nginx-ingress-36721569, Cluster IP: 10.47.255.248, External IPs ['207.208.148.60'] 2026-02-05 01:39:45,251 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-36721569 2026-02-05 01:39:45,251 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_LoadbalancerDatabaseList?x=cee26fe5-db03-42a2-bc6f-9d085a4aa43c 2026-02-05 01:39:45,262 - INFO - Ingress ctest-nginx-ingress-36721569 with uuid cee26fe5-db03-42a2-bc6f-9d085a4aa43c found in kube manager 2026-02-05 01:39:45,262 - INFO - Ingress ctest-nginx-ingress-36721569 verification passed 2026-02-05 01:39:45,270 - DEBUG - Pod ctest-busybox-pod-61901625 not present 2026-02-05 01:39:45,270 - INFO - Creating Pod ctest-busybox-pod-61901625 2026-02-05 01:39:45,298 - DEBUG - Pod : ctest-busybox-pod-61901625 UUID is 24c7e04e-ca90-4664-8ba6-0018bb6cfea4 2026-02-05 01:39:45,308 - DEBUG - Pod ctest-nginx-pod-38910120 not in running state.Currently in Pending 2026-02-05 01:39:50,316 - DEBUG - Pod ctest-nginx-pod-38910120 not in running state.Currently in Pending 2026-02-05 01:39:55,324 - INFO - Pod ctest-nginx-pod-38910120 is in running state.Got IP 10.47.255.250 2026-02-05 01:39:55,377 - DEBUG - Pod ctest-nginx-pod-38910120 has vmi 88987ce4-0233-11f1-9e52-0242d3cf2fbf 2026-02-05 01:39:55,377 - INFO - Verified pod ctest-nginx-pod-38910120 in contrail-api 2026-02-05 01:39:55,377 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-05 01:39:55,388 - DEBUG - VMI 88987ce4-0233-11f1-9e52-0242d3cf2fbf is active in agent 10.0.0.33 2026-02-05 01:39:55,388 - INFO - Verified Pod ctest-nginx-pod-38910120 in agent 10.0.0.33 2026-02-05 01:39:55,388 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=cba5bdf6-98be-4696-91a5-825d0e98e56b 2026-02-05 01:39:55,392 - INFO - Pod ctest-nginx-pod-38910120 with uuid cba5bdf6-98be-4696-91a5-825d0e98e56b found in kube manager 2026-02-05 01:39:55,392 - INFO - Pod ctest-nginx-pod-38910120 verification passed 2026-02-05 01:39:55,482 - DEBUG - [Pod ctest-nginx-pod-38910120] Cmd: echo ctest-nginx-pod-38910120 > /usr/share/nginx/html/index.html, Output: 2026-02-05 01:39:55,490 - INFO - Pod ctest-nginx-pod-55172250 is in running state.Got IP 10.47.255.249 2026-02-05 01:39:55,543 - DEBUG - Pod ctest-nginx-pod-55172250 has vmi 89481b0e-0233-11f1-9e52-0242d3cf2fbf 2026-02-05 01:39:55,544 - INFO - Verified pod ctest-nginx-pod-55172250 in contrail-api 2026-02-05 01:39:55,544 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-05 01:39:55,555 - DEBUG - VMI 89481b0e-0233-11f1-9e52-0242d3cf2fbf is active in agent 10.0.0.33 2026-02-05 01:39:55,555 - INFO - Verified Pod ctest-nginx-pod-55172250 in agent 10.0.0.33 2026-02-05 01:39:55,555 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=67c9264b-7a7a-4aad-ac90-51dfc3f570cc 2026-02-05 01:39:55,558 - INFO - Pod ctest-nginx-pod-55172250 with uuid 67c9264b-7a7a-4aad-ac90-51dfc3f570cc found in kube manager 2026-02-05 01:39:55,558 - INFO - Pod ctest-nginx-pod-55172250 verification passed 2026-02-05 01:39:55,638 - DEBUG - [Pod ctest-nginx-pod-55172250] Cmd: echo ctest-nginx-pod-55172250 > /usr/share/nginx/html/index.html, Output: 2026-02-05 01:39:55,645 - DEBUG - Pod ctest-busybox-pod-61901625 not in running state.Currently in Pending 2026-02-05 01:40:00,657 - INFO - Pod ctest-busybox-pod-61901625 is in running state.Got IP 10.47.255.247 2026-02-05 01:40:00,714 - DEBUG - Pod ctest-busybox-pod-61901625 has vmi 8ea45978-0233-11f1-9e52-0242d3cf2fbf 2026-02-05 01:40:00,714 - INFO - Verified pod ctest-busybox-pod-61901625 in contrail-api 2026-02-05 01:40:00,714 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-05 01:40:00,727 - DEBUG - VMI 8ea45978-0233-11f1-9e52-0242d3cf2fbf is active in agent 10.0.0.33 2026-02-05 01:40:00,727 - INFO - Verified Pod ctest-busybox-pod-61901625 in agent 10.0.0.33 2026-02-05 01:40:00,727 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=24c7e04e-ca90-4664-8ba6-0018bb6cfea4 2026-02-05 01:40:00,732 - INFO - Pod ctest-busybox-pod-61901625 with uuid 24c7e04e-ca90-4664-8ba6-0018bb6cfea4 found in kube manager 2026-02-05 01:40:00,732 - INFO - Pod ctest-busybox-pod-61901625 verification passed 2026-02-05 01:40:00,842 - DEBUG - [Pod ctest-busybox-pod-61901625] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-38910120 2026-02-05 01:40:00,842 - DEBUG - [Pod Pod ctest-busybox-pod-61901625] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-02-05 01:40:00,842 - DEBUG - [Pod Pod ctest-busybox-pod-61901625] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-38910120 2026-02-05 01:40:00,943 - DEBUG - [Pod ctest-busybox-pod-61901625] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-55172250 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-05 01:40:00,944 - DEBUG - [Pod Pod ctest-busybox-pod-61901625] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-02-05 01:40:00,944 - DEBUG - [Pod Pod ctest-busybox-pod-61901625] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-55172250 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-05 01:40:00,944 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-38910120': 1, 'ctest-nginx-pod-55172250': 1} 2026-02-05 01:40:00,944 - WARNING - Nothing to delete parallely 2026-02-05 01:40:00,944 - INFO - Deleting pod default:ctest-busybox-pod-61901625 2026-02-05 01:40:00,964 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=24c7e04e-ca90-4664-8ba6-0018bb6cfea4 2026-02-05 01:40:00,977 - WARNING - Pod uuid 24c7e04e-ca90-4664-8ba6-0018bb6cfea4 is still seen in agent 10.0.0.33 VM list 2026-02-05 01:40:05,978 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=24c7e04e-ca90-4664-8ba6-0018bb6cfea4 2026-02-05 01:40:05,984 - WARNING - Pod uuid 24c7e04e-ca90-4664-8ba6-0018bb6cfea4 is still seen in agent 10.0.0.33 VM list 2026-02-05 01:40:10,985 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=24c7e04e-ca90-4664-8ba6-0018bb6cfea4 2026-02-05 01:40:10,991 - WARNING - Pod uuid 24c7e04e-ca90-4664-8ba6-0018bb6cfea4 is still seen in agent 10.0.0.33 VM list 2026-02-05 01:40:15,992 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=24c7e04e-ca90-4664-8ba6-0018bb6cfea4 2026-02-05 01:40:15,998 - WARNING - Pod uuid 24c7e04e-ca90-4664-8ba6-0018bb6cfea4 is still seen in agent 10.0.0.33 VM list 2026-02-05 01:40:20,999 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=24c7e04e-ca90-4664-8ba6-0018bb6cfea4 2026-02-05 01:40:21,005 - WARNING - Pod uuid 24c7e04e-ca90-4664-8ba6-0018bb6cfea4 is still seen in agent 10.0.0.33 VM list 2026-02-05 01:40:26,006 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=24c7e04e-ca90-4664-8ba6-0018bb6cfea4 2026-02-05 01:40:26,013 - WARNING - Pod uuid 24c7e04e-ca90-4664-8ba6-0018bb6cfea4 is still seen in agent 10.0.0.33 VM list 2026-02-05 01:40:31,014 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=24c7e04e-ca90-4664-8ba6-0018bb6cfea4 2026-02-05 01:40:31,024 - WARNING - Pod uuid 24c7e04e-ca90-4664-8ba6-0018bb6cfea4 is still seen in agent 10.0.0.33 VM list 2026-02-05 01:40:36,025 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=24c7e04e-ca90-4664-8ba6-0018bb6cfea4 2026-02-05 01:40:36,031 - DEBUG - Pod 24c7e04e-ca90-4664-8ba6-0018bb6cfea4 is not in agent 10.0.0.33 VM list 2026-02-05 01:40:36,031 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:8ea45978-0233-11f1-9e52-0242d3cf2fbf 2026-02-05 01:40:36,036 - DEBUG - VMI 8ea45978-0233-11f1-9e52-0242d3cf2fbf is removed from agent 10.0.0.33 2026-02-05 01:40:36,036 - INFO - Verified that pod ctest-busybox-pod-61901625 is removed in agent 2026-02-05 01:40:36,036 - INFO - Deleting Ingress : ctest-nginx-ingress-36721569 2026-02-05 01:40:36,046 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_LoadbalancerDatabaseList?x=cee26fe5-db03-42a2-bc6f-9d085a4aa43c 2026-02-05 01:40:36,050 - ERROR - Ingress ctest-nginx-ingress-36721569 with uuid cee26fe5-db03-42a2-bc6f-9d085a4aa43c still found in kube manager 2026-02-05 01:40:37,051 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_LoadbalancerDatabaseList?x=cee26fe5-db03-42a2-bc6f-9d085a4aa43c 2026-02-05 01:40:37,057 - ERROR - Ingress ctest-nginx-ingress-36721569 with uuid cee26fe5-db03-42a2-bc6f-9d085a4aa43c still found in kube manager 2026-02-05 01:40:38,058 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_LoadbalancerDatabaseList?x=cee26fe5-db03-42a2-bc6f-9d085a4aa43c 2026-02-05 01:40:38,062 - INFO - Ingress ctest-nginx-ingress-36721569 with uuid cee26fe5-db03-42a2-bc6f-9d085a4aa43c deleted successfully from kube manager 2026-02-05 01:40:38,062 - INFO - Verifications on Ingress %s cleanup passed 2026-02-05 01:40:38,062 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-02-05 01:40:38,192 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-05 01:40:38,199 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-02-05 01:40:38,200 - INFO - Deleting VN __public__ 2026-02-05 01:40:38,371 - DEBUG - Requesting: http://10.0.0.33:8082/routing-instance/d1728205-7240-4f90-b9c8-857437b00f5f 2026-02-05 01:40:38,376 - DEBUG - Response Code: 404 2026-02-05 01:40:38,376 - DEBUG - Requesting: http://10.0.0.33:8082/domains 2026-02-05 01:40:38,397 - DEBUG - Requesting: http://10.0.0.33:8082/domain/2738dc43-b459-4bf5-951e-64e8b58fb46c 2026-02-05 01:40:38,407 - DEBUG - Requesting: http://10.0.0.33:8082/project/b0f57af3-d71e-404d-9ab8-ef54213610e1 2026-02-05 01:40:38,475 - INFO - Validated that VN __public__ is not found in API Server 2026-02-05 01:40:38,476 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-02-05 01:40:38,481 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-05 01:40:38,486 - DEBUG - VN __public__ is not present in Agent 10.0.0.33 2026-02-05 01:40:38,486 - INFO - Validated that VN __public__ is not in any agent 2026-02-05 01:40:38,487 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-05 01:40:38,493 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-05 01:40:38,501 - DEBUG - Requesting: http://10.0.0.33:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-05 01:40:38,508 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-02-05 01:40:38,509 - INFO - Deleting pod default:ctest-nginx-pod-55172250 2026-02-05 01:40:38,528 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=67c9264b-7a7a-4aad-ac90-51dfc3f570cc 2026-02-05 01:40:38,539 - WARNING - Pod uuid 67c9264b-7a7a-4aad-ac90-51dfc3f570cc is still seen in agent 10.0.0.33 VM list 2026-02-05 01:40:43,540 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=67c9264b-7a7a-4aad-ac90-51dfc3f570cc 2026-02-05 01:40:43,545 - DEBUG - Pod 67c9264b-7a7a-4aad-ac90-51dfc3f570cc is not in agent 10.0.0.33 VM list 2026-02-05 01:40:43,545 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:89481b0e-0233-11f1-9e52-0242d3cf2fbf 2026-02-05 01:40:43,551 - DEBUG - VMI 89481b0e-0233-11f1-9e52-0242d3cf2fbf is removed from agent 10.0.0.33 2026-02-05 01:40:43,551 - INFO - Verified that pod ctest-nginx-pod-55172250 is removed in agent 2026-02-05 01:40:43,551 - INFO - Deleting pod default:ctest-nginx-pod-38910120 2026-02-05 01:40:43,573 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=cba5bdf6-98be-4696-91a5-825d0e98e56b 2026-02-05 01:40:43,581 - WARNING - Pod uuid cba5bdf6-98be-4696-91a5-825d0e98e56b is still seen in agent 10.0.0.33 VM list 2026-02-05 01:40:48,582 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=cba5bdf6-98be-4696-91a5-825d0e98e56b 2026-02-05 01:40:48,589 - DEBUG - Pod cba5bdf6-98be-4696-91a5-825d0e98e56b is not in agent 10.0.0.33 VM list 2026-02-05 01:40:48,589 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:88987ce4-0233-11f1-9e52-0242d3cf2fbf 2026-02-05 01:40:48,594 - DEBUG - VMI 88987ce4-0233-11f1-9e52-0242d3cf2fbf is removed from agent 10.0.0.33 2026-02-05 01:40:48,594 - INFO - Verified that pod ctest-nginx-pod-38910120 is removed in agent 2026-02-05 01:40:48,594 - INFO - Deleting service : ctest-nginx-svc-44055259 2026-02-05 01:40:49,160 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-05 01:40:49,165 - DEBUG - No XMPP flaps were noticed during the test