2026-03-26 16:46:56,780 - DEBUG - [10.0.0.38]: Running cmd : hostname 2026-03-26 16:46:57,030 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-386-1 2026-03-26 16:46:57,030 - DEBUG - [10.0.0.38]: Running cmd : hostname -f 2026-03-26 16:46:57,160 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-386-1.cluster.local 2026-03-26 16:46:57,161 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-03-26 16:46:57,377 - DEBUG - Output : NAMES contrail_test_dhpdJ8zWt k8s_autoscaler_dns-autoscaler-595558c478-bftzh_kube-system_542923b5-a89f-4874-84b7-b48be2f90055_0 k8s_coredns_coredns-77f7cc69db-dv9g9_kube-system_f179d455-29e6-4620-9e91-5918fe231011_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-6d9tw_kube-system_9388dfe1-d2bd-49c3-8089-1cb86b0b5f09_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-6d9tw_kube-system_9388dfe1-d2bd-49c3-8089-1cb86b0b5f09_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-6d9tw_kube-system_9388dfe1-d2bd-49c3-8089-1cb86b0b5f09_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-bhfps_kube-system_d6e9d139-0803-4354-943c-3b8b7f7275fc_0 k8s_opensdn-controller-control-named_opensdn-controller-control-bhfps_kube-system_d6e9d139-0803-4354-943c-3b8b7f7275fc_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-9wwgc_kube-system_19ca7973-9678-47ba-acb0-5b19b929d108_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-hnzgv_kube-system_e97adf25-b8ae-481a-9e18-6bf4f41ee6c4_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-hnzgv_kube-system_e97adf25-b8ae-481a-9e18-6bf4f41ee6c4_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-l9fk9_kube-system_9a9352c4-7ec1-4136-ad47-b694845ac448_0 k8s_opensdn-analytics-collector_opensdn-analytics-l9fk9_kube-system_9a9352c4-7ec1-4136-ad47-b694845ac448_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-6d9tw_kube-system_9388dfe1-d2bd-49c3-8089-1cb86b0b5f09_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-bhfps_kube-system_d6e9d139-0803-4354-943c-3b8b7f7275fc_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-892q9_kube-system_af8a5f35-9480-4082-91fb-2fa7add4f5da_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-892q9_kube-system_af8a5f35-9480-4082-91fb-2fa7add4f5da_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-slg7t_kube-system_086c4d21-a10a-4a28-a274-a2ab85392ac1_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-9wwgc_kube-system_19ca7973-9678-47ba-acb0-5b19b929d108_0 k8s_kafka_opensdn-analytics-alarm-hnzgv_kube-system_e97adf25-b8ae-481a-9e18-6bf4f41ee6c4_0 k8s_opensdn-analytics-api_opensdn-analytics-l9fk9_kube-system_9a9352c4-7ec1-4136-ad47-b694845ac448_0 k8s_opensdn-controller-config-api_opensdn-controller-config-6d9tw_kube-system_9388dfe1-d2bd-49c3-8089-1cb86b0b5f09_0 k8s_opensdn-controller-control_opensdn-controller-control-bhfps_kube-system_d6e9d139-0803-4354-943c-3b8b7f7275fc_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-892q9_kube-system_af8a5f35-9480-4082-91fb-2fa7add4f5da_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-slg7t_kube-system_086c4d21-a10a-4a28-a274-a2ab85392ac1_0 k8s_redis_redis-5j5h7_kube-system_b6425989-d258-41bd-81a9-e1d5bdb9f4a8_0 k8s_rabbitmq_rabbitmq-7fntw_kube-system_8cc15ceb-f627-4532-a58d-24b429c72a44_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-dgwht_kube-system_eebc37d1-d2db-4d2a-894d-2aaa1e6ed990_0 k8s_opensdn-configdb_opensdn-configdb-dgwht_kube-system_eebc37d1-d2db-4d2a-894d-2aaa1e6ed990_0 k8s_config-zookeeper_config-zookeeper-mtqrk_kube-system_05ad2a69-f38c-49c3-8550-516c1211f348_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-9wwgc_kube-system_19ca7973-9678-47ba-acb0-5b19b929d108_0 k8s_opensdn-agent-nodemgr_opensdn-agent-hqpxq_kube-system_47bb9441-10fc-454d-98d2-468354930bd5_0 k8s_opensdn-vrouter-agent_opensdn-agent-hqpxq_kube-system_47bb9441-10fc-454d-98d2-468354930bd5_0 k8s_opensdn-kube-manager_opensdn-kube-manager-swrbb_kube-system_a459b6de-16d4-4553-b349-e010dcbe3ca8_0 k8s_kube-proxy_kube-proxy-vwvgm_kube-system_3e79d227-ef9b-44af-93ab-d47f267a3def_0 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-386-1_kube-system_abb04e82bb18ee283fd1775618acb0ce_1 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-386-1_kube-system_446bb4f0cb1f88e499eeeb158dd6d9a1_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-386-1_kube-system_e1ee71ab13141d147e76b40cd58efeb8_0 2026-03-26 16:46:57,378 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-03-26 16:46:57,515 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.3/32 10.96.0.1/32 10.0.0.38/24 2026-03-26 16:46:57,515 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}' 2026-03-26 16:46:57,635 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-386-1.cluster.local 2026-03-26 16:46:57,636 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-03-26 16:46:57,758 - DEBUG - Output : 10.0.0.38/24 2026-03-26 16:46:58,157 - INFO - Using existing project ['default-domain', 'k8s-default'](d1023a13-abc9-4711-9534-c0ab3068f8e4) 2026-03-26 16:46:58,311 - INFO - Using existing project ['default-domain', 'k8s-default'](d1023a13-abc9-4711-9534-c0ab3068f8e4) 2026-03-26 16:46:58,441 - INFO - ================================================================================ 2026-03-26 16:46:58,442 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-03-26 16:46:58,442 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-03-26 16:46:58,718 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-26 16:46:58,725 - DEBUG - Nothing to compare xmpp stats {'10.0.0.38': {'10.0.0.38': '2'}} with 2026-03-26 16:46:58,725 - INFO - Initial checks done. Running the testcase now 2026-03-26 16:46:58,725 - INFO - 2026-03-26 16:46:58,729 - INFO - Namespace default already exists 2026-03-26 16:46:58,730 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-swrbb_kube-system_a459b6de-16d4-4553-b349-e010dcbe3ca8_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-03-26 16:46:58,957 - DEBUG - Output : cluster_project={} 2026-03-26 16:46:59,044 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-03-26 16:46:59,045 - DEBUG - Project uuid in Contrail is d1023a13-abc9-4711-9534-c0ab3068f8e4 2026-03-26 16:46:59,045 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-swrbb_kube-system_a459b6de-16d4-4553-b349-e010dcbe3ca8_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-03-26 16:46:59,202 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-03-26 16:46:59,202 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_NamespaceDatabaseList?namespace_uuid=795635bb-2768-451c-b240-da94383e4c29 2026-03-26 16:46:59,206 - INFO - Namespace default with uuid 795635bb-2768-451c-b240-da94383e4c29 found in kube manager 2026-03-26 16:46:59,206 - INFO - Namespace default verification passed 2026-03-26 16:46:59,209 - DEBUG - Service ctest-nginx-svc-17310968 not present 2026-03-26 16:46:59,210 - INFO - Creating service ctest-nginx-svc-17310968 2026-03-26 16:46:59,227 - INFO - Created Service ctest-nginx-svc-17310968 2026-03-26 16:46:59,230 - DEBUG - Pod ctest-nginx-pod-07446363 not present 2026-03-26 16:46:59,231 - INFO - Creating Pod ctest-nginx-pod-07446363 2026-03-26 16:46:59,272 - DEBUG - Pod : ctest-nginx-pod-07446363 UUID is c4c7a03c-1975-4b8d-9470-de35dafac696 2026-03-26 16:46:59,276 - DEBUG - Pod ctest-nginx-pod-03011809 not present 2026-03-26 16:46:59,277 - INFO - Creating Pod ctest-nginx-pod-03011809 2026-03-26 16:46:59,314 - DEBUG - Pod : ctest-nginx-pod-03011809 UUID is 96cf380e-dab8-4660-ab07-ccbfce24fd65 2026-03-26 16:46:59,585 - INFO - Created VN __public__, UUID :6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:46:59,822 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-26 16:46:59,827 - DEBUG - Requesting: http://10.0.0.38:8082/domain/7eb72ecf-2daf-4587-8a2c-abfffea88397 2026-03-26 16:46:59,850 - DEBUG - Requesting: http://10.0.0.38:8082/project/d1023a13-abc9-4711-9534-c0ab3068f8e4 2026-03-26 16:46:59,934 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:46:59,944 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:46:59,951 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/c2219c28-82b1-4da4-b4b2-162fe61f958d 2026-03-26 16:46:59,957 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/c2219c28-82b1-4da4-b4b2-162fe61f958d 2026-03-26 16:46:59,962 - DEBUG - Route Targets: [] 2026-03-26 16:46:59,962 - DEBUG - RT names not yet present for VN __public__ 2026-03-26 16:47:04,963 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-26 16:47:04,969 - DEBUG - Requesting: http://10.0.0.38:8082/domain/7eb72ecf-2daf-4587-8a2c-abfffea88397 2026-03-26 16:47:04,975 - DEBUG - Requesting: http://10.0.0.38:8082/project/d1023a13-abc9-4711-9534-c0ab3068f8e4 2026-03-26 16:47:05,014 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:47:05,021 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:47:05,028 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/c2219c28-82b1-4da4-b4b2-162fe61f958d 2026-03-26 16:47:05,036 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/c2219c28-82b1-4da4-b4b2-162fe61f958d 2026-03-26 16:47:05,043 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/4c71413d-e32d-439f-bba1-23ae86cb3d7f 2026-03-26 16:47:05,048 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-26 16:47:05,048 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:47:05,056 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/c2219c28-82b1-4da4-b4b2-162fe61f958d 2026-03-26 16:47:05,066 - INFO - Verified VN network id 11 for VN 6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:47:05,066 - INFO - Verifications in API Server for VN __public__ passed 2026-03-26 16:47:05,067 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:47:05,074 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/c2219c28-82b1-4da4-b4b2-162fe61f958d 2026-03-26 16:47:05,080 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/c2219c28-82b1-4da4-b4b2-162fe61f958d 2026-03-26 16:47:05,085 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/4c71413d-e32d-439f-bba1-23ae86cb3d7f 2026-03-26 16:47:05,089 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-26 16:47:05,095 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7734697576164050957', 'uuid-lslong': '10586415522179153394'}, 'enable': 'true', 'created': '2026-03-26T16:46:59', 'last-modified': '2026-03-26T16:46:59', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.393624'} 2026-03-26 16:47:05,095 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-26 16:47:05,104 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-26 16:47:05,104 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-03-26 16:47:05,104 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-03-26 16:47:05,104 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-26 16:47:05,109 - DEBUG - Requesting: http://10.0.0.38:8082/domain/7eb72ecf-2daf-4587-8a2c-abfffea88397 2026-03-26 16:47:05,114 - DEBUG - Requesting: http://10.0.0.38:8082/project/d1023a13-abc9-4711-9534-c0ab3068f8e4 2026-03-26 16:47:05,150 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:47:05,160 - DEBUG - =>VN __public__ has no policy to be verified 2026-03-26 16:47:05,160 - DEBUG - Verifying the vn in opserver 2026-03-26 16:47:05,160 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.38 2026-03-26 16:47:05,160 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2026-03-26 16:47:05,164 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-03-26 16:47:05,164 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-03-26 16:47:05,164 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-26 16:47:05,169 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-26 16:47:05,174 - DEBUG - Do not have enough data to verify VN in agent 2026-03-26 16:47:05,174 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-26 16:47:05,178 - DEBUG - VRF ids for VN __public__: {} 2026-03-26 16:47:05,338 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-03-26 16:47:05,553 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-26 16:47:05,558 - DEBUG - Requesting: http://10.0.0.38:8082/domain/7eb72ecf-2daf-4587-8a2c-abfffea88397 2026-03-26 16:47:05,567 - DEBUG - Requesting: http://10.0.0.38:8082/project/d1023a13-abc9-4711-9534-c0ab3068f8e4 2026-03-26 16:47:05,606 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:47:05,617 - DEBUG - Requesting: http://10.0.0.38:8082/floating-ip-pool/544fee0c-e085-49c9-bc5d-61c9efb9427e 2026-03-26 16:47:05,621 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-26 16:47:05,625 - DEBUG - Requesting: http://10.0.0.38:8082/domain/7eb72ecf-2daf-4587-8a2c-abfffea88397 2026-03-26 16:47:05,630 - DEBUG - Requesting: http://10.0.0.38:8082/project/d1023a13-abc9-4711-9534-c0ab3068f8e4 2026-03-26 16:47:05,657 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b572a77-2055-480d-92ea-80d4042b7df2 2026-03-26 16:47:05,665 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-03-26 16:47:05,665 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-03-26 16:47:05,672 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-26 16:47:05,680 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-03-26 16:47:05,680 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-03-26 16:47:05,684 - DEBUG - Ingress ctest-nginx-ingress-76863225 not present 2026-03-26 16:47:05,684 - INFO - Creating Ingress ctest-nginx-ingress-76863225 2026-03-26 16:47:05,690 - INFO - Created Ingress ctest-nginx-ingress-76863225 2026-03-26 16:47:05,695 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-76863225 2026-03-26 16:47:08,702 - DEBUG - For Ingress ctest-nginx-ingress-76863225, Cluster IP: 10.47.255.248, External IPs ['170.54.84.60'] 2026-03-26 16:47:08,702 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-76863225 2026-03-26 16:47:08,702 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=914063e9-72ff-4ae0-9326-378b7cf33ef7 2026-03-26 16:47:08,711 - INFO - Ingress ctest-nginx-ingress-76863225 with uuid 914063e9-72ff-4ae0-9326-378b7cf33ef7 found in kube manager 2026-03-26 16:47:08,711 - INFO - Ingress ctest-nginx-ingress-76863225 verification passed 2026-03-26 16:47:08,715 - DEBUG - Pod ctest-busybox-pod-32482226 not present 2026-03-26 16:47:08,715 - INFO - Creating Pod ctest-busybox-pod-32482226 2026-03-26 16:47:08,741 - DEBUG - Pod : ctest-busybox-pod-32482226 UUID is ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee 2026-03-26 16:47:08,758 - DEBUG - Pod ctest-nginx-pod-07446363 not in running state.Currently in Pending 2026-03-26 16:47:13,767 - DEBUG - Pod ctest-nginx-pod-07446363 not in running state.Currently in Pending 2026-03-26 16:47:18,775 - INFO - Pod ctest-nginx-pod-07446363 is in running state.Got IP 10.47.255.250 2026-03-26 16:47:18,828 - DEBUG - Pod ctest-nginx-pod-07446363 has vmi 69ac1a6c-2933-11f1-9b3b-02425790ceca 2026-03-26 16:47:18,828 - INFO - Verified pod ctest-nginx-pod-07446363 in contrail-api 2026-03-26 16:47:18,828 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-26 16:47:18,837 - DEBUG - VMI 69ac1a6c-2933-11f1-9b3b-02425790ceca is active in agent 10.0.0.38 2026-03-26 16:47:18,837 - INFO - Verified Pod ctest-nginx-pod-07446363 in agent 10.0.0.38 2026-03-26 16:47:18,837 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=c4c7a03c-1975-4b8d-9470-de35dafac696 2026-03-26 16:47:18,841 - INFO - Pod ctest-nginx-pod-07446363 with uuid c4c7a03c-1975-4b8d-9470-de35dafac696 found in kube manager 2026-03-26 16:47:18,841 - INFO - Pod ctest-nginx-pod-07446363 verification passed 2026-03-26 16:47:18,997 - DEBUG - [Pod ctest-nginx-pod-07446363] Cmd: echo ctest-nginx-pod-07446363 > /usr/share/nginx/html/index.html, Output: 2026-03-26 16:47:19,008 - INFO - Pod ctest-nginx-pod-03011809 is in running state.Got IP 10.47.255.249 2026-03-26 16:47:19,064 - DEBUG - Pod ctest-nginx-pod-03011809 has vmi 6a5be636-2933-11f1-9b3b-02425790ceca 2026-03-26 16:47:19,064 - INFO - Verified pod ctest-nginx-pod-03011809 in contrail-api 2026-03-26 16:47:19,064 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-26 16:47:19,074 - DEBUG - VMI 6a5be636-2933-11f1-9b3b-02425790ceca is active in agent 10.0.0.38 2026-03-26 16:47:19,074 - INFO - Verified Pod ctest-nginx-pod-03011809 in agent 10.0.0.38 2026-03-26 16:47:19,074 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=96cf380e-dab8-4660-ab07-ccbfce24fd65 2026-03-26 16:47:19,079 - INFO - Pod ctest-nginx-pod-03011809 with uuid 96cf380e-dab8-4660-ab07-ccbfce24fd65 found in kube manager 2026-03-26 16:47:19,079 - INFO - Pod ctest-nginx-pod-03011809 verification passed 2026-03-26 16:47:19,171 - DEBUG - [Pod ctest-nginx-pod-03011809] Cmd: echo ctest-nginx-pod-03011809 > /usr/share/nginx/html/index.html, Output: 2026-03-26 16:47:19,182 - DEBUG - Pod ctest-busybox-pod-32482226 not in running state.Currently in Pending 2026-03-26 16:47:24,189 - INFO - Pod ctest-busybox-pod-32482226 is in running state.Got IP 10.47.255.247 2026-03-26 16:47:24,241 - DEBUG - Pod ctest-busybox-pod-32482226 has vmi 6fe301e8-2933-11f1-9b3b-02425790ceca 2026-03-26 16:47:24,241 - INFO - Verified pod ctest-busybox-pod-32482226 in contrail-api 2026-03-26 16:47:24,241 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-26 16:47:24,250 - DEBUG - VMI 6fe301e8-2933-11f1-9b3b-02425790ceca is active in agent 10.0.0.38 2026-03-26 16:47:24,251 - INFO - Verified Pod ctest-busybox-pod-32482226 in agent 10.0.0.38 2026-03-26 16:47:24,251 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee 2026-03-26 16:47:24,255 - INFO - Pod ctest-busybox-pod-32482226 with uuid ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee found in kube manager 2026-03-26 16:47:24,255 - INFO - Pod ctest-busybox-pod-32482226 verification passed 2026-03-26 16:47:24,327 - DEBUG - [Pod ctest-busybox-pod-32482226] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-03011809 Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-26 16:47:24,327 - DEBUG - [Pod Pod ctest-busybox-pod-32482226] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-26 16:47:24,327 - DEBUG - [Pod Pod ctest-busybox-pod-32482226] Cmd output: ctest-nginx-pod-03011809 Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-26 16:47:24,420 - DEBUG - [Pod ctest-busybox-pod-32482226] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-03011809 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-26 16:47:24,421 - DEBUG - [Pod Pod ctest-busybox-pod-32482226] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-26 16:47:24,421 - DEBUG - [Pod Pod ctest-busybox-pod-32482226] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-03011809 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-26 16:47:24,494 - DEBUG - [Pod ctest-busybox-pod-32482226] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-07446363 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-26 16:47:24,494 - DEBUG - [Pod Pod ctest-busybox-pod-32482226] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-26 16:47:24,494 - DEBUG - [Pod Pod ctest-busybox-pod-32482226] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-07446363 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-26 16:47:24,494 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-07446363': 1, 'ctest-nginx-pod-03011809': 2} 2026-03-26 16:47:24,495 - WARNING - Nothing to delete parallely 2026-03-26 16:47:24,495 - INFO - Deleting pod default:ctest-busybox-pod-32482226 2026-03-26 16:47:24,506 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee 2026-03-26 16:47:24,512 - WARNING - Pod uuid ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee is still seen in agent 10.0.0.38 VM list 2026-03-26 16:47:29,512 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee 2026-03-26 16:47:29,517 - WARNING - Pod uuid ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee is still seen in agent 10.0.0.38 VM list 2026-03-26 16:47:34,519 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee 2026-03-26 16:47:34,524 - WARNING - Pod uuid ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee is still seen in agent 10.0.0.38 VM list 2026-03-26 16:47:39,524 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee 2026-03-26 16:47:39,530 - WARNING - Pod uuid ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee is still seen in agent 10.0.0.38 VM list 2026-03-26 16:47:44,531 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee 2026-03-26 16:47:44,538 - WARNING - Pod uuid ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee is still seen in agent 10.0.0.38 VM list 2026-03-26 16:47:49,539 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee 2026-03-26 16:47:49,545 - WARNING - Pod uuid ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee is still seen in agent 10.0.0.38 VM list 2026-03-26 16:47:54,545 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee 2026-03-26 16:47:54,551 - WARNING - Pod uuid ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee is still seen in agent 10.0.0.38 VM list 2026-03-26 16:47:59,553 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee 2026-03-26 16:47:59,557 - DEBUG - Pod ea337a1f-5b4b-407b-9b6d-2ca9bae2ccee is not in agent 10.0.0.38 VM list 2026-03-26 16:47:59,557 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6fe301e8-2933-11f1-9b3b-02425790ceca 2026-03-26 16:47:59,564 - DEBUG - VMI 6fe301e8-2933-11f1-9b3b-02425790ceca is removed from agent 10.0.0.38 2026-03-26 16:47:59,564 - INFO - Verified that pod ctest-busybox-pod-32482226 is removed in agent 2026-03-26 16:47:59,564 - INFO - Deleting Ingress : ctest-nginx-ingress-76863225 2026-03-26 16:47:59,573 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=914063e9-72ff-4ae0-9326-378b7cf33ef7 2026-03-26 16:47:59,579 - ERROR - Ingress ctest-nginx-ingress-76863225 with uuid 914063e9-72ff-4ae0-9326-378b7cf33ef7 still found in kube manager 2026-03-26 16:48:00,580 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=914063e9-72ff-4ae0-9326-378b7cf33ef7 2026-03-26 16:48:00,600 - ERROR - Ingress ctest-nginx-ingress-76863225 with uuid 914063e9-72ff-4ae0-9326-378b7cf33ef7 still found in kube manager 2026-03-26 16:48:01,601 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=914063e9-72ff-4ae0-9326-378b7cf33ef7 2026-03-26 16:48:01,606 - INFO - Ingress ctest-nginx-ingress-76863225 with uuid 914063e9-72ff-4ae0-9326-378b7cf33ef7 deleted successfully from kube manager 2026-03-26 16:48:01,606 - INFO - Verifications on Ingress %s cleanup passed 2026-03-26 16:48:01,606 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-03-26 16:48:01,744 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-26 16:48:01,748 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-03-26 16:48:01,748 - INFO - Deleting VN __public__ 2026-03-26 16:48:01,824 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/c2219c28-82b1-4da4-b4b2-162fe61f958d 2026-03-26 16:48:01,830 - DEBUG - Response Code: 404 2026-03-26 16:48:01,830 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-26 16:48:01,835 - DEBUG - Requesting: http://10.0.0.38:8082/domain/7eb72ecf-2daf-4587-8a2c-abfffea88397 2026-03-26 16:48:01,842 - DEBUG - Requesting: http://10.0.0.38:8082/project/d1023a13-abc9-4711-9534-c0ab3068f8e4 2026-03-26 16:48:01,885 - INFO - Validated that VN __public__ is not found in API Server 2026-03-26 16:48:01,885 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-03-26 16:48:01,889 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-26 16:48:01,894 - DEBUG - VN __public__ is not present in Agent 10.0.0.38 2026-03-26 16:48:01,894 - INFO - Validated that VN __public__ is not in any agent 2026-03-26 16:48:01,894 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-26 16:48:01,898 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-26 16:48:01,903 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-26 16:48:01,907 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-03-26 16:48:01,908 - INFO - Deleting pod default:ctest-nginx-pod-03011809 2026-03-26 16:48:01,918 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=96cf380e-dab8-4660-ab07-ccbfce24fd65 2026-03-26 16:48:01,928 - WARNING - Pod uuid 96cf380e-dab8-4660-ab07-ccbfce24fd65 is still seen in agent 10.0.0.38 VM list 2026-03-26 16:48:06,929 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=96cf380e-dab8-4660-ab07-ccbfce24fd65 2026-03-26 16:48:06,934 - DEBUG - Pod 96cf380e-dab8-4660-ab07-ccbfce24fd65 is not in agent 10.0.0.38 VM list 2026-03-26 16:48:06,934 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6a5be636-2933-11f1-9b3b-02425790ceca 2026-03-26 16:48:06,938 - DEBUG - VMI 6a5be636-2933-11f1-9b3b-02425790ceca is removed from agent 10.0.0.38 2026-03-26 16:48:06,938 - INFO - Verified that pod ctest-nginx-pod-03011809 is removed in agent 2026-03-26 16:48:06,939 - INFO - Deleting pod default:ctest-nginx-pod-07446363 2026-03-26 16:48:06,961 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=c4c7a03c-1975-4b8d-9470-de35dafac696 2026-03-26 16:48:06,971 - WARNING - Pod uuid c4c7a03c-1975-4b8d-9470-de35dafac696 is still seen in agent 10.0.0.38 VM list 2026-03-26 16:48:11,972 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=c4c7a03c-1975-4b8d-9470-de35dafac696 2026-03-26 16:48:11,977 - DEBUG - Pod c4c7a03c-1975-4b8d-9470-de35dafac696 is not in agent 10.0.0.38 VM list 2026-03-26 16:48:11,977 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:69ac1a6c-2933-11f1-9b3b-02425790ceca 2026-03-26 16:48:11,983 - DEBUG - VMI 69ac1a6c-2933-11f1-9b3b-02425790ceca is removed from agent 10.0.0.38 2026-03-26 16:48:11,983 - INFO - Verified that pod ctest-nginx-pod-07446363 is removed in agent 2026-03-26 16:48:11,983 - INFO - Deleting service : ctest-nginx-svc-17310968 2026-03-26 16:48:12,538 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-26 16:48:12,543 - DEBUG - No XMPP flaps were noticed during the test