2025-12-29 01:43:04,019 - DEBUG - [10.0.0.249]: Running cmd : hostname 2025-12-29 01:43:04,275 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-283-1 2025-12-29 01:43:04,276 - DEBUG - [10.0.0.249]: Running cmd : hostname -f 2025-12-29 01:43:04,412 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-283-1.cluster.local 2025-12-29 01:43:04,413 - DEBUG - [10.0.0.249]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-12-29 01:43:04,581 - DEBUG - Output : NAMES contrail_test_x7us4yrtJ k8s_autoscaler_dns-autoscaler-595558c478-nrwpr_kube-system_b94e6400-7c9f-483e-865b-bfb89752982a_0 k8s_coredns_coredns-77f7cc69db-d7bnh_kube-system_d67304cc-14ba-428c-acf3-83a8a643a79f_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-zjn5v_kube-system_63f44ea1-62e3-4d63-bcbc-a84e4d8113b5_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-zjn5v_kube-system_63f44ea1-62e3-4d63-bcbc-a84e4d8113b5_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-rrvc5_kube-system_d4f80565-c057-4f94-bf48-80904ff8db1f_0 k8s_opensdn-controller-control-named_opensdn-controller-control-rrvc5_kube-system_d4f80565-c057-4f94-bf48-80904ff8db1f_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-zjn5v_kube-system_63f44ea1-62e3-4d63-bcbc-a84e4d8113b5_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-kxg4x_kube-system_37c7be80-e040-43fb-8b9a-f587d5b65dea_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-fvrld_kube-system_6d53f289-6dd5-44b5-8289-8f0ddf58df56_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-fvrld_kube-system_6d53f289-6dd5-44b5-8289-8f0ddf58df56_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-cvqkg_kube-system_aa2f9fe7-d94c-4ec9-8418-90608a879dde_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-gspps_kube-system_5b2e046e-ae19-4296-be14-96ac59368247_0 k8s_opensdn-analytics-collector_opensdn-analytics-gspps_kube-system_5b2e046e-ae19-4296-be14-96ac59368247_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-rrvc5_kube-system_d4f80565-c057-4f94-bf48-80904ff8db1f_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-zjn5v_kube-system_63f44ea1-62e3-4d63-bcbc-a84e4d8113b5_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-kxg4x_kube-system_37c7be80-e040-43fb-8b9a-f587d5b65dea_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-gdm5t_kube-system_3a80a555-f145-4e0c-9d70-e9fd01810910_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-gdm5t_kube-system_3a80a555-f145-4e0c-9d70-e9fd01810910_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-6lb2v_kube-system_70794e2f-e353-41a9-99a2-b03beb71c827_0 k8s_opensdn-agent-nodemgr_opensdn-agent-klvd5_kube-system_ab9611f5-d767-43f6-9e91-1c8b7563ba1c_0 k8s_kafka_opensdn-analytics-alarm-fvrld_kube-system_6d53f289-6dd5-44b5-8289-8f0ddf58df56_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-cvqkg_kube-system_aa2f9fe7-d94c-4ec9-8418-90608a879dde_0 k8s_opensdn-analytics-api_opensdn-analytics-gspps_kube-system_5b2e046e-ae19-4296-be14-96ac59368247_0 k8s_opensdn-controller-control_opensdn-controller-control-rrvc5_kube-system_d4f80565-c057-4f94-bf48-80904ff8db1f_0 k8s_opensdn-controller-config-api_opensdn-controller-config-zjn5v_kube-system_63f44ea1-62e3-4d63-bcbc-a84e4d8113b5_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-kxg4x_kube-system_37c7be80-e040-43fb-8b9a-f587d5b65dea_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-gdm5t_kube-system_3a80a555-f145-4e0c-9d70-e9fd01810910_0 k8s_config-zookeeper_config-zookeeper-f6vg6_kube-system_07fd6353-e88c-44fb-975c-41c0a2e6bf3f_0 k8s_redis_redis-69642_kube-system_f8dae132-378f-4c8b-a7c9-1a047431237b_0 k8s_rabbitmq_rabbitmq-7jnzc_kube-system_aa9486a8-8f41-4336-bbd2-8c36d412abb0_0 k8s_opensdn-configdb_opensdn-configdb-6lb2v_kube-system_70794e2f-e353-41a9-99a2-b03beb71c827_0 k8s_opensdn-vrouter-agent_opensdn-agent-klvd5_kube-system_ab9611f5-d767-43f6-9e91-1c8b7563ba1c_0 k8s_opensdn-kube-manager_opensdn-kube-manager-rlzrt_kube-system_3e95bb7b-301a-46a5-be5a-924e1cb7c41b_0 k8s_kube-proxy_kube-proxy-lqssr_kube-system_d9e90864-77dd-4ca6-baa6-e36d55f2688d_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-283-1_kube-system_35d5541a73184b8da6b1025a1fadaada_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-283-1_kube-system_f9e357421b3e545be9e5467785676841_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-283-1_kube-system_8848769d725ba26ab5f9efbc107bbc42_0 2025-12-29 01:43:04,581 - DEBUG - [10.0.0.249]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-12-29 01:43:04,699 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.249/24 2025-12-29 01:43:04,700 - DEBUG - [10.0.0.249]: Running cmd : getent hosts 10.0.0.249 | head -n 1 | awk '{print $2}' 2025-12-29 01:43:04,824 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-283-1.cluster.local 2025-12-29 01:43:04,824 - DEBUG - [10.0.0.249]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-12-29 01:43:04,941 - DEBUG - Output : 10.0.0.249/24 2025-12-29 01:43:05,340 - INFO - Using existing project ['default-domain', 'k8s-default'](ff41a364-2327-413d-8db8-c426934782df) 2025-12-29 01:43:05,498 - INFO - Using existing project ['default-domain', 'k8s-default'](ff41a364-2327-413d-8db8-c426934782df) 2025-12-29 01:43:05,685 - INFO - ================================================================================ 2025-12-29 01:43:05,686 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-12-29 01:43:05,686 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-12-29 01:43:05,996 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq? 2025-12-29 01:43:06,005 - DEBUG - Nothing to compare xmpp stats {'10.0.0.249': {'10.0.0.249': '2'}} with 2025-12-29 01:43:06,006 - INFO - Initial checks done. Running the testcase now 2025-12-29 01:43:06,006 - INFO - 2025-12-29 01:43:06,012 - INFO - Namespace default already exists 2025-12-29 01:43:06,013 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-rlzrt_kube-system_3e95bb7b-301a-46a5-be5a-924e1cb7c41b_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-12-29 01:43:06,185 - DEBUG - Output : cluster_project={} 2025-12-29 01:43:06,262 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-12-29 01:43:06,263 - DEBUG - Project uuid in Contrail is ff41a364-2327-413d-8db8-c426934782df 2025-12-29 01:43:06,263 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-rlzrt_kube-system_3e95bb7b-301a-46a5-be5a-924e1cb7c41b_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-12-29 01:43:06,443 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-12-29 01:43:06,443 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_NamespaceDatabaseList?namespace_uuid=5ef13ba3-1883-451b-a345-8300ac1a930b 2025-12-29 01:43:06,447 - INFO - Namespace default with uuid 5ef13ba3-1883-451b-a345-8300ac1a930b found in kube manager 2025-12-29 01:43:06,447 - INFO - Namespace default verification passed 2025-12-29 01:43:06,451 - DEBUG - Service ctest-nginx-svc-53224397 not present 2025-12-29 01:43:06,451 - INFO - Creating service ctest-nginx-svc-53224397 2025-12-29 01:43:06,470 - INFO - Created Service ctest-nginx-svc-53224397 2025-12-29 01:43:06,475 - DEBUG - Pod ctest-nginx-pod-69780482 not present 2025-12-29 01:43:06,475 - INFO - Creating Pod ctest-nginx-pod-69780482 2025-12-29 01:43:06,509 - DEBUG - Pod : ctest-nginx-pod-69780482 UUID is f6a0bbd6-56ec-474c-827b-3bb15662064d 2025-12-29 01:43:06,512 - DEBUG - Pod ctest-nginx-pod-51407310 not present 2025-12-29 01:43:06,513 - INFO - Creating Pod ctest-nginx-pod-51407310 2025-12-29 01:43:06,547 - DEBUG - Pod : ctest-nginx-pod-51407310 UUID is 92d3494c-be2c-4919-9369-4c2029a6555b 2025-12-29 01:43:06,832 - INFO - Created VN __public__, UUID :0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:07,067 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-29 01:43:07,074 - DEBUG - Requesting: http://10.0.0.249:8082/domain/aaef326e-392f-4f79-b268-bc9bda0caa7d 2025-12-29 01:43:07,159 - DEBUG - Requesting: http://10.0.0.249:8082/project/ff41a364-2327-413d-8db8-c426934782df 2025-12-29 01:43:07,258 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:07,288 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:07,298 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/269847ce-9631-4f45-8c81-7502346f0762 2025-12-29 01:43:07,306 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/269847ce-9631-4f45-8c81-7502346f0762 2025-12-29 01:43:07,316 - DEBUG - Route Targets: [] 2025-12-29 01:43:07,316 - DEBUG - RT names not yet present for VN __public__ 2025-12-29 01:43:12,316 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-29 01:43:12,321 - DEBUG - Requesting: http://10.0.0.249:8082/domain/aaef326e-392f-4f79-b268-bc9bda0caa7d 2025-12-29 01:43:12,329 - DEBUG - Requesting: http://10.0.0.249:8082/project/ff41a364-2327-413d-8db8-c426934782df 2025-12-29 01:43:12,373 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:12,381 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:12,389 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/269847ce-9631-4f45-8c81-7502346f0762 2025-12-29 01:43:12,395 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/269847ce-9631-4f45-8c81-7502346f0762 2025-12-29 01:43:12,401 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/4ace2fee-cc09-44e8-a59a-77b0e12cac71 2025-12-29 01:43:12,407 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-29 01:43:12,407 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:12,415 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/269847ce-9631-4f45-8c81-7502346f0762 2025-12-29 01:43:12,426 - INFO - Verified VN network id 11 for VN 0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:12,427 - INFO - Verifications in API Server for VN __public__ passed 2025-12-29 01:43:12,427 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:12,436 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/269847ce-9631-4f45-8c81-7502346f0762 2025-12-29 01:43:12,442 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/269847ce-9631-4f45-8c81-7502346f0762 2025-12-29 01:43:12,447 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/4ace2fee-cc09-44e8-a59a-77b0e12cac71 2025-12-29 01:43:12,453 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-12-29 01:43:12,458 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '986213203125879694', 'uuid-lslong': '13395569922416479484'}, 'enable': 'true', 'created': '2025-12-29T01:43:06', 'last-modified': '2025-12-29T01:43:06', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.473968'} 2025-12-29 01:43:12,458 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-29 01:43:12,467 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-29 01:43:12,467 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-12-29 01:43:12,467 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-12-29 01:43:12,467 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-29 01:43:12,472 - DEBUG - Requesting: http://10.0.0.249:8082/domain/aaef326e-392f-4f79-b268-bc9bda0caa7d 2025-12-29 01:43:12,478 - DEBUG - Requesting: http://10.0.0.249:8082/project/ff41a364-2327-413d-8db8-c426934782df 2025-12-29 01:43:12,511 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:12,519 - DEBUG - =>VN __public__ has no policy to be verified 2025-12-29 01:43:12,519 - DEBUG - Verifying the vn in opserver 2025-12-29 01:43:12,519 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.249 2025-12-29 01:43:12,519 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2025-12-29 01:43:12,541 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-12-29 01:43:12,541 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-12-29 01:43:12,541 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-29 01:43:12,550 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-29 01:43:12,556 - DEBUG - Do not have enough data to verify VN in agent 2025-12-29 01:43:12,556 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-29 01:43:12,560 - DEBUG - VRF ids for VN __public__: {} 2025-12-29 01:43:12,716 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-12-29 01:43:12,921 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-29 01:43:12,927 - DEBUG - Requesting: http://10.0.0.249:8082/domain/aaef326e-392f-4f79-b268-bc9bda0caa7d 2025-12-29 01:43:12,933 - DEBUG - Requesting: http://10.0.0.249:8082/project/ff41a364-2327-413d-8db8-c426934782df 2025-12-29 01:43:12,961 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:12,999 - DEBUG - Requesting: http://10.0.0.249:8082/floating-ip-pool/1e4b3053-5407-45d3-b6ad-41f18a0f987f 2025-12-29 01:43:13,008 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-29 01:43:13,027 - DEBUG - Requesting: http://10.0.0.249:8082/domain/aaef326e-392f-4f79-b268-bc9bda0caa7d 2025-12-29 01:43:13,047 - DEBUG - Requesting: http://10.0.0.249:8082/project/ff41a364-2327-413d-8db8-c426934782df 2025-12-29 01:43:13,088 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/0dafbbae-dd1e-438e-b9e6-9fdf258a8cfc 2025-12-29 01:43:13,119 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-12-29 01:43:13,119 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-12-29 01:43:13,134 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-12-29 01:43:13,149 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-12-29 01:43:13,149 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-12-29 01:43:13,154 - DEBUG - Ingress ctest-nginx-ingress-29942160 not present 2025-12-29 01:43:13,154 - INFO - Creating Ingress ctest-nginx-ingress-29942160 2025-12-29 01:43:13,164 - INFO - Created Ingress ctest-nginx-ingress-29942160 2025-12-29 01:43:13,169 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-29942160 2025-12-29 01:43:16,174 - DEBUG - For Ingress ctest-nginx-ingress-29942160, Cluster IP: 10.47.255.248, External IPs ['9.0.4.252'] 2025-12-29 01:43:16,175 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-29942160 2025-12-29 01:43:16,175 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=edaacbd7-1b95-466b-aed1-08a01e270ab9 2025-12-29 01:43:16,187 - INFO - Ingress ctest-nginx-ingress-29942160 with uuid edaacbd7-1b95-466b-aed1-08a01e270ab9 found in kube manager 2025-12-29 01:43:16,187 - INFO - Ingress ctest-nginx-ingress-29942160 verification passed 2025-12-29 01:43:16,190 - DEBUG - Pod ctest-busybox-pod-46657550 not present 2025-12-29 01:43:16,191 - INFO - Creating Pod ctest-busybox-pod-46657550 2025-12-29 01:43:16,217 - DEBUG - Pod : ctest-busybox-pod-46657550 UUID is f0ef67d4-e813-4d81-8aa9-c520d8085d8e 2025-12-29 01:43:16,227 - DEBUG - Pod ctest-nginx-pod-69780482 not in running state.Currently in Pending 2025-12-29 01:43:21,235 - INFO - Pod ctest-nginx-pod-69780482 is in running state.Got IP 10.47.255.250 2025-12-29 01:43:21,437 - DEBUG - Pod ctest-nginx-pod-69780482 has vmi baa3b450-e457-11f0-b9fc-024251aeaa93 2025-12-29 01:43:21,437 - INFO - Verified pod ctest-nginx-pod-69780482 in contrail-api 2025-12-29 01:43:21,437 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-12-29 01:43:21,452 - DEBUG - VMI baa3b450-e457-11f0-b9fc-024251aeaa93 is active in agent 10.0.0.249 2025-12-29 01:43:21,452 - INFO - Verified Pod ctest-nginx-pod-69780482 in agent 10.0.0.249 2025-12-29 01:43:21,452 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=f6a0bbd6-56ec-474c-827b-3bb15662064d 2025-12-29 01:43:21,466 - INFO - Pod ctest-nginx-pod-69780482 with uuid f6a0bbd6-56ec-474c-827b-3bb15662064d found in kube manager 2025-12-29 01:43:21,466 - INFO - Pod ctest-nginx-pod-69780482 verification passed 2025-12-29 01:43:21,639 - DEBUG - [Pod ctest-nginx-pod-69780482] Cmd: echo ctest-nginx-pod-69780482 > /usr/share/nginx/html/index.html, Output: 2025-12-29 01:43:21,648 - INFO - Pod ctest-nginx-pod-51407310 is in running state.Got IP 10.47.255.249 2025-12-29 01:43:21,711 - DEBUG - Pod ctest-nginx-pod-51407310 has vmi bb533e52-e457-11f0-b9fc-024251aeaa93 2025-12-29 01:43:21,711 - INFO - Verified pod ctest-nginx-pod-51407310 in contrail-api 2025-12-29 01:43:21,712 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-12-29 01:43:21,720 - DEBUG - VMI bb533e52-e457-11f0-b9fc-024251aeaa93 is active in agent 10.0.0.249 2025-12-29 01:43:21,720 - INFO - Verified Pod ctest-nginx-pod-51407310 in agent 10.0.0.249 2025-12-29 01:43:21,720 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=92d3494c-be2c-4919-9369-4c2029a6555b 2025-12-29 01:43:21,725 - INFO - Pod ctest-nginx-pod-51407310 with uuid 92d3494c-be2c-4919-9369-4c2029a6555b found in kube manager 2025-12-29 01:43:21,725 - INFO - Pod ctest-nginx-pod-51407310 verification passed 2025-12-29 01:43:21,824 - DEBUG - [Pod ctest-nginx-pod-51407310] Cmd: echo ctest-nginx-pod-51407310 > /usr/share/nginx/html/index.html, Output: 2025-12-29 01:43:21,832 - DEBUG - Pod ctest-busybox-pod-46657550 not in running state.Currently in Pending 2025-12-29 01:43:26,840 - DEBUG - Pod ctest-busybox-pod-46657550 not in running state.Currently in Pending 2025-12-29 01:43:31,847 - INFO - Pod ctest-busybox-pod-46657550 is in running state.Got IP 10.47.255.247 2025-12-29 01:43:31,900 - DEBUG - Pod ctest-busybox-pod-46657550 has vmi c0e4c6ba-e457-11f0-b9fc-024251aeaa93 2025-12-29 01:43:31,900 - INFO - Verified pod ctest-busybox-pod-46657550 in contrail-api 2025-12-29 01:43:31,900 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-12-29 01:43:31,908 - DEBUG - VMI c0e4c6ba-e457-11f0-b9fc-024251aeaa93 is active in agent 10.0.0.249 2025-12-29 01:43:31,908 - INFO - Verified Pod ctest-busybox-pod-46657550 in agent 10.0.0.249 2025-12-29 01:43:31,909 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=f0ef67d4-e813-4d81-8aa9-c520d8085d8e 2025-12-29 01:43:31,913 - INFO - Pod ctest-busybox-pod-46657550 with uuid f0ef67d4-e813-4d81-8aa9-c520d8085d8e found in kube manager 2025-12-29 01:43:31,913 - INFO - Pod ctest-busybox-pod-46657550 verification passed 2025-12-29 01:43:31,986 - DEBUG - [Pod ctest-busybox-pod-46657550] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-69780482 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-12-29 01:43:31,986 - DEBUG - [Pod Pod ctest-busybox-pod-46657550] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-12-29 01:43:31,986 - DEBUG - [Pod Pod ctest-busybox-pod-46657550] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-69780482 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-12-29 01:43:32,065 - DEBUG - [Pod ctest-busybox-pod-46657550] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-69780482 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-12-29 01:43:32,065 - DEBUG - [Pod Pod ctest-busybox-pod-46657550] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-12-29 01:43:32,065 - DEBUG - [Pod Pod ctest-busybox-pod-46657550] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-69780482 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-12-29 01:43:32,139 - DEBUG - [Pod ctest-busybox-pod-46657550] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-69780482 2025-12-29 01:43:32,139 - DEBUG - [Pod Pod ctest-busybox-pod-46657550] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-12-29 01:43:32,139 - DEBUG - [Pod Pod ctest-busybox-pod-46657550] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-69780482 2025-12-29 01:43:32,207 - DEBUG - [Pod ctest-busybox-pod-46657550] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-69780482 2025-12-29 01:43:32,208 - DEBUG - [Pod Pod ctest-busybox-pod-46657550] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-12-29 01:43:32,208 - DEBUG - [Pod Pod ctest-busybox-pod-46657550] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-69780482 2025-12-29 01:43:32,282 - DEBUG - [Pod ctest-busybox-pod-46657550] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-51407310 2025-12-29 01:43:32,282 - DEBUG - [Pod Pod ctest-busybox-pod-46657550] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-12-29 01:43:32,282 - DEBUG - [Pod Pod ctest-busybox-pod-46657550] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-51407310 2025-12-29 01:43:32,282 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-69780482': 4, 'ctest-nginx-pod-51407310': 1} 2025-12-29 01:43:32,282 - WARNING - Nothing to delete parallely 2025-12-29 01:43:32,282 - INFO - Deleting pod default:ctest-busybox-pod-46657550 2025-12-29 01:43:32,298 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f0ef67d4-e813-4d81-8aa9-c520d8085d8e 2025-12-29 01:43:32,303 - WARNING - Pod uuid f0ef67d4-e813-4d81-8aa9-c520d8085d8e is still seen in agent 10.0.0.249 VM list 2025-12-29 01:43:37,304 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f0ef67d4-e813-4d81-8aa9-c520d8085d8e 2025-12-29 01:43:37,309 - WARNING - Pod uuid f0ef67d4-e813-4d81-8aa9-c520d8085d8e is still seen in agent 10.0.0.249 VM list 2025-12-29 01:43:42,310 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f0ef67d4-e813-4d81-8aa9-c520d8085d8e 2025-12-29 01:43:42,317 - WARNING - Pod uuid f0ef67d4-e813-4d81-8aa9-c520d8085d8e is still seen in agent 10.0.0.249 VM list 2025-12-29 01:43:47,318 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f0ef67d4-e813-4d81-8aa9-c520d8085d8e 2025-12-29 01:43:47,323 - WARNING - Pod uuid f0ef67d4-e813-4d81-8aa9-c520d8085d8e is still seen in agent 10.0.0.249 VM list 2025-12-29 01:43:52,324 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f0ef67d4-e813-4d81-8aa9-c520d8085d8e 2025-12-29 01:43:52,331 - WARNING - Pod uuid f0ef67d4-e813-4d81-8aa9-c520d8085d8e is still seen in agent 10.0.0.249 VM list 2025-12-29 01:43:57,332 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f0ef67d4-e813-4d81-8aa9-c520d8085d8e 2025-12-29 01:43:57,337 - WARNING - Pod uuid f0ef67d4-e813-4d81-8aa9-c520d8085d8e is still seen in agent 10.0.0.249 VM list 2025-12-29 01:44:02,338 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f0ef67d4-e813-4d81-8aa9-c520d8085d8e 2025-12-29 01:44:02,344 - WARNING - Pod uuid f0ef67d4-e813-4d81-8aa9-c520d8085d8e is still seen in agent 10.0.0.249 VM list 2025-12-29 01:44:07,345 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f0ef67d4-e813-4d81-8aa9-c520d8085d8e 2025-12-29 01:44:07,350 - DEBUG - Pod f0ef67d4-e813-4d81-8aa9-c520d8085d8e is not in agent 10.0.0.249 VM list 2025-12-29 01:44:07,350 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:c0e4c6ba-e457-11f0-b9fc-024251aeaa93 2025-12-29 01:44:07,355 - DEBUG - VMI c0e4c6ba-e457-11f0-b9fc-024251aeaa93 is removed from agent 10.0.0.249 2025-12-29 01:44:07,355 - INFO - Verified that pod ctest-busybox-pod-46657550 is removed in agent 2025-12-29 01:44:07,356 - INFO - Deleting Ingress : ctest-nginx-ingress-29942160 2025-12-29 01:44:07,364 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=edaacbd7-1b95-466b-aed1-08a01e270ab9 2025-12-29 01:44:07,370 - ERROR - Ingress ctest-nginx-ingress-29942160 with uuid edaacbd7-1b95-466b-aed1-08a01e270ab9 still found in kube manager 2025-12-29 01:44:08,371 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=edaacbd7-1b95-466b-aed1-08a01e270ab9 2025-12-29 01:44:08,379 - ERROR - Ingress ctest-nginx-ingress-29942160 with uuid edaacbd7-1b95-466b-aed1-08a01e270ab9 still found in kube manager 2025-12-29 01:44:09,379 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=edaacbd7-1b95-466b-aed1-08a01e270ab9 2025-12-29 01:44:09,383 - INFO - Ingress ctest-nginx-ingress-29942160 with uuid edaacbd7-1b95-466b-aed1-08a01e270ab9 deleted successfully from kube manager 2025-12-29 01:44:09,383 - INFO - Verifications on Ingress %s cleanup passed 2025-12-29 01:44:09,383 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-12-29 01:44:09,516 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-12-29 01:44:09,572 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-12-29 01:44:09,573 - INFO - Deleting VN __public__ 2025-12-29 01:44:09,789 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/269847ce-9631-4f45-8c81-7502346f0762 2025-12-29 01:44:09,794 - DEBUG - Response Code: 404 2025-12-29 01:44:09,794 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-29 01:44:09,804 - DEBUG - Requesting: http://10.0.0.249:8082/domain/aaef326e-392f-4f79-b268-bc9bda0caa7d 2025-12-29 01:44:09,813 - DEBUG - Requesting: http://10.0.0.249:8082/project/ff41a364-2327-413d-8db8-c426934782df 2025-12-29 01:44:09,851 - INFO - Validated that VN __public__ is not found in API Server 2025-12-29 01:44:09,851 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-12-29 01:44:09,858 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-29 01:44:09,865 - DEBUG - VN __public__ is not present in Agent 10.0.0.249 2025-12-29 01:44:09,865 - INFO - Validated that VN __public__ is not in any agent 2025-12-29 01:44:09,865 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-29 01:44:09,870 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-29 01:44:09,875 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-12-29 01:44:09,879 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-12-29 01:44:09,880 - INFO - Deleting pod default:ctest-nginx-pod-51407310 2025-12-29 01:44:09,889 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=92d3494c-be2c-4919-9369-4c2029a6555b 2025-12-29 01:44:09,897 - WARNING - Pod uuid 92d3494c-be2c-4919-9369-4c2029a6555b is still seen in agent 10.0.0.249 VM list 2025-12-29 01:44:14,898 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=92d3494c-be2c-4919-9369-4c2029a6555b 2025-12-29 01:44:14,904 - DEBUG - Pod 92d3494c-be2c-4919-9369-4c2029a6555b is not in agent 10.0.0.249 VM list 2025-12-29 01:44:14,904 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:bb533e52-e457-11f0-b9fc-024251aeaa93 2025-12-29 01:44:14,911 - DEBUG - VMI bb533e52-e457-11f0-b9fc-024251aeaa93 is removed from agent 10.0.0.249 2025-12-29 01:44:14,911 - INFO - Verified that pod ctest-nginx-pod-51407310 is removed in agent 2025-12-29 01:44:14,911 - INFO - Deleting pod default:ctest-nginx-pod-69780482 2025-12-29 01:44:14,926 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f6a0bbd6-56ec-474c-827b-3bb15662064d 2025-12-29 01:44:14,933 - WARNING - Pod uuid f6a0bbd6-56ec-474c-827b-3bb15662064d is still seen in agent 10.0.0.249 VM list 2025-12-29 01:44:19,934 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f6a0bbd6-56ec-474c-827b-3bb15662064d 2025-12-29 01:44:19,941 - DEBUG - Pod f6a0bbd6-56ec-474c-827b-3bb15662064d is not in agent 10.0.0.249 VM list 2025-12-29 01:44:19,941 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:baa3b450-e457-11f0-b9fc-024251aeaa93 2025-12-29 01:44:19,949 - DEBUG - VMI baa3b450-e457-11f0-b9fc-024251aeaa93 is removed from agent 10.0.0.249 2025-12-29 01:44:19,949 - INFO - Verified that pod ctest-nginx-pod-69780482 is removed in agent 2025-12-29 01:44:19,949 - INFO - Deleting service : ctest-nginx-svc-53224397 2025-12-29 01:44:20,509 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq? 2025-12-29 01:44:20,514 - DEBUG - No XMPP flaps were noticed during the test