2025-08-25 01:48:36,249 - DEBUG - [10.0.0.48]: Running cmd : hostname 2025-08-25 01:48:36,636 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-239-1 2025-08-25 01:48:36,636 - DEBUG - [10.0.0.48]: Running cmd : hostname -f 2025-08-25 01:48:36,809 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local 2025-08-25 01:48:36,809 - DEBUG - [10.0.0.48]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-25 01:48:37,075 - DEBUG - Output : NAMES contrail_test_p8Xwo6QIU k8s_autoscaler_dns-autoscaler-595558c478-xmlhm_kube-system_e512aeb7-54b1-4eca-ac6d-20369c039624_0 k8s_coredns_coredns-77f7cc69db-mv7kg_kube-system_73cef897-6ba1-4c9d-bbcc-5918dfcbbf7a_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-6hvdd_kube-system_fb5f2411-6117-4b75-af82-b758bd6085ed_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-6hvdd_kube-system_fb5f2411-6117-4b75-af82-b758bd6085ed_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-s2lbh_kube-system_9830b8b2-d911-439f-ba0d-b85a73016a8d_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-fzrzd_kube-system_6aa7ba7d-8631-47a2-9565-71c0debfc408_0 k8s_opensdn-controller-control-named_opensdn-controller-control-fzrzd_kube-system_6aa7ba7d-8631-47a2-9565-71c0debfc408_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-vd26s_kube-system_4fc86e86-6c75-4b78-bfe5-6c2733c3bbaa_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-6hvdd_kube-system_fb5f2411-6117-4b75-af82-b758bd6085ed_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-vd26s_kube-system_4fc86e86-6c75-4b78-bfe5-6c2733c3bbaa_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-s2lbh_kube-system_9830b8b2-d911-439f-ba0d-b85a73016a8d_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-l778d_kube-system_fdb9a398-de69-4ed0-b15a-d66950ae9772_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-l778d_kube-system_fdb9a398-de69-4ed0-b15a-d66950ae9772_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-psq8x_kube-system_5a108377-2ba7-48b4-a2e0-9b362e1a1a98_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-8zhvq_kube-system_abb9b68e-d57a-4714-a703-5f718d90767c_0 k8s_opensdn-analytics-collector_opensdn-analytics-psq8x_kube-system_5a108377-2ba7-48b4-a2e0-9b362e1a1a98_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-fzrzd_kube-system_6aa7ba7d-8631-47a2-9565-71c0debfc408_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-6hvdd_kube-system_fb5f2411-6117-4b75-af82-b758bd6085ed_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-vd26s_kube-system_4fc86e86-6c75-4b78-bfe5-6c2733c3bbaa_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-s2lbh_kube-system_9830b8b2-d911-439f-ba0d-b85a73016a8d_0 k8s_kafka_opensdn-analytics-alarm-l778d_kube-system_fdb9a398-de69-4ed0-b15a-d66950ae9772_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-8zhvq_kube-system_abb9b68e-d57a-4714-a703-5f718d90767c_0 k8s_opensdn-analytics-api_opensdn-analytics-psq8x_kube-system_5a108377-2ba7-48b4-a2e0-9b362e1a1a98_0 k8s_opensdn-controller-control_opensdn-controller-control-fzrzd_kube-system_6aa7ba7d-8631-47a2-9565-71c0debfc408_0 k8s_opensdn-controller-config-api_opensdn-controller-config-6hvdd_kube-system_fb5f2411-6117-4b75-af82-b758bd6085ed_0 k8s_redis_redis-5hnpt_kube-system_0e2552ea-aa9f-4b08-b11e-ebfe8516c100_0 k8s_config-zookeeper_config-zookeeper-dcn77_kube-system_1eda39bf-aa6f-40ac-9192-8215d09514ee_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-wd9ql_kube-system_61c953a1-c858-4475-a58e-c91e96ab93ec_0 k8s_opensdn-configdb_opensdn-configdb-wd9ql_kube-system_61c953a1-c858-4475-a58e-c91e96ab93ec_0 k8s_rabbitmq_rabbitmq-tqghd_kube-system_fe002629-60d2-4e77-ba50-4fc1ba9fd037_0 k8s_opensdn-agent-nodemgr_opensdn-agent-74qbb_kube-system_5b1ead20-437f-4998-bd62-6a140ace8c56_0 k8s_opensdn-vrouter-agent_opensdn-agent-74qbb_kube-system_5b1ead20-437f-4998-bd62-6a140ace8c56_0 k8s_opensdn-kube-manager_opensdn-kube-manager-grr8v_kube-system_35751fb9-ae10-44be-8f88-2a1be4b73ef6_0 k8s_kube-proxy_kube-proxy-7s7sp_kube-system_d593880f-b858-4b5b-b09e-a85e6fa811b2_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-239-1_kube-system_5789cbb26075386791279f453df4341e_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-239-1_kube-system_1b4878b2d9d6da12032156d4dfebde4c_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-239-1_kube-system_70d7195a10fbae9a508eb5c629bd54d6_0 2025-08-25 01:48:37,075 - DEBUG - [10.0.0.48]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-25 01:48:37,286 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.3/32 10.96.0.1/32 10.0.0.48/24 2025-08-25 01:48:37,287 - DEBUG - [10.0.0.48]: Running cmd : getent hosts 10.0.0.48 | head -n 1 | awk '{print $2}' 2025-08-25 01:48:37,436 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local 2025-08-25 01:48:37,436 - DEBUG - [10.0.0.48]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-25 01:48:37,628 - DEBUG - Output : 10.0.0.48/24 2025-08-25 01:48:38,133 - INFO - Using existing project ['default-domain', 'k8s-default'](e7ebff81-059b-4396-840c-6ce2a40f909f) 2025-08-25 01:48:38,353 - INFO - Using existing project ['default-domain', 'k8s-default'](e7ebff81-059b-4396-840c-6ce2a40f909f) 2025-08-25 01:48:38,523 - INFO - ================================================================================ 2025-08-25 01:48:38,524 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-08-25 01:48:38,525 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-08-25 01:48:38,921 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-25 01:48:38,930 - DEBUG - Nothing to compare xmpp stats {'10.0.0.48': {'10.0.0.48': '3'}} with 2025-08-25 01:48:38,930 - INFO - Initial checks done. Running the testcase now 2025-08-25 01:48:38,930 - INFO - 2025-08-25 01:48:38,935 - INFO - Namespace default already exists 2025-08-25 01:48:38,936 - DEBUG - [10.0.0.48]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-grr8v_kube-system_35751fb9-ae10-44be-8f88-2a1be4b73ef6_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-08-25 01:48:39,151 - DEBUG - Output : cluster_project={} 2025-08-25 01:48:39,280 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-08-25 01:48:39,280 - DEBUG - Project uuid in Contrail is e7ebff81-059b-4396-840c-6ce2a40f909f 2025-08-25 01:48:39,281 - DEBUG - [10.0.0.48]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-grr8v_kube-system_35751fb9-ae10-44be-8f88-2a1be4b73ef6_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-08-25 01:48:39,507 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-08-25 01:48:39,507 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_NamespaceDatabaseList?namespace_uuid=3aa1f0be-928c-48c3-9a17-57748f4e40ac 2025-08-25 01:48:39,513 - INFO - Namespace default with uuid 3aa1f0be-928c-48c3-9a17-57748f4e40ac found in kube manager 2025-08-25 01:48:39,513 - INFO - Namespace default verification passed 2025-08-25 01:48:39,517 - DEBUG - Service ctest-nginx-svc-02211479 not present 2025-08-25 01:48:39,518 - INFO - Creating service ctest-nginx-svc-02211479 2025-08-25 01:48:39,538 - INFO - Created Service ctest-nginx-svc-02211479 2025-08-25 01:48:39,542 - DEBUG - Pod ctest-nginx-pod-96045388 not present 2025-08-25 01:48:39,542 - INFO - Creating Pod ctest-nginx-pod-96045388 2025-08-25 01:48:39,572 - DEBUG - Pod : ctest-nginx-pod-96045388 UUID is 20a8e29a-36cb-4b45-8596-4632ae5de0fb 2025-08-25 01:48:39,583 - DEBUG - Pod ctest-nginx-pod-65143093 not present 2025-08-25 01:48:39,583 - INFO - Creating Pod ctest-nginx-pod-65143093 2025-08-25 01:48:39,607 - DEBUG - Pod : ctest-nginx-pod-65143093 UUID is 9bf56c28-b1f6-45da-9017-298e1672e102 2025-08-25 01:48:40,178 - INFO - Created VN __public__, UUID :d5a132f9-4473-4b77-95fd-93d8f851b554 2025-08-25 01:48:40,779 - DEBUG - Requesting: http://10.0.0.48:8082/domains 2025-08-25 01:48:40,876 - DEBUG - Requesting: http://10.0.0.48:8082/domain/a32a9a80-558e-4500-92e4-2d929868eed5 2025-08-25 01:48:40,990 - DEBUG - Requesting: http://10.0.0.48:8082/project/e7ebff81-059b-4396-840c-6ce2a40f909f 2025-08-25 01:48:41,120 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/d5a132f9-4473-4b77-95fd-93d8f851b554 2025-08-25 01:48:41,139 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/d5a132f9-4473-4b77-95fd-93d8f851b554 2025-08-25 01:48:41,192 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/2b5fca39-565d-4e12-a112-64fb89e0dad0 2025-08-25 01:48:41,207 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/2b5fca39-565d-4e12-a112-64fb89e0dad0 2025-08-25 01:48:41,220 - DEBUG - Requesting: http://10.0.0.48:8082/route-target/00aafcb7-146c-49e2-80e3-36070e739ac6 2025-08-25 01:48:41,231 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-25 01:48:41,231 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/d5a132f9-4473-4b77-95fd-93d8f851b554 2025-08-25 01:48:41,244 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/2b5fca39-565d-4e12-a112-64fb89e0dad0 2025-08-25 01:48:41,267 - INFO - Verified VN network id 12 for VN d5a132f9-4473-4b77-95fd-93d8f851b554 2025-08-25 01:48:41,267 - INFO - Verifications in API Server for VN __public__ passed 2025-08-25 01:48:41,268 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/d5a132f9-4473-4b77-95fd-93d8f851b554 2025-08-25 01:48:41,287 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/2b5fca39-565d-4e12-a112-64fb89e0dad0 2025-08-25 01:48:41,302 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/2b5fca39-565d-4e12-a112-64fb89e0dad0 2025-08-25 01:48:41,313 - DEBUG - Requesting: http://10.0.0.48:8082/route-target/00aafcb7-146c-49e2-80e3-36070e739ac6 2025-08-25 01:48:41,323 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-08-25 01:48:41,339 - DEBUG - Control-node 10.0.0.48 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15393641047505718135', 'uuid-lslong': '10807957240847381844'}, 'enable': 'true', 'created': '2025-08-25T01:48:39', 'last-modified': '2025-08-25T01:48:40', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.771199'} 2025-08-25 01:48:41,339 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-25 01:48:41,358 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-25 01:48:41,358 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-08-25 01:48:41,359 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-08-25 01:48:41,359 - DEBUG - Requesting: http://10.0.0.48:8082/domains 2025-08-25 01:48:41,370 - DEBUG - Requesting: http://10.0.0.48:8082/domain/a32a9a80-558e-4500-92e4-2d929868eed5 2025-08-25 01:48:41,381 - DEBUG - Requesting: http://10.0.0.48:8082/project/e7ebff81-059b-4396-840c-6ce2a40f909f 2025-08-25 01:48:41,414 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/d5a132f9-4473-4b77-95fd-93d8f851b554 2025-08-25 01:48:41,436 - DEBUG - =>VN __public__ has no policy to be verified 2025-08-25 01:48:41,436 - DEBUG - Verifying the vn in opserver 2025-08-25 01:48:41,436 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.48 2025-08-25 01:48:41,436 - DEBUG - Requesting: http://10.0.0.48:8081/analytics/uves/virtual-networks 2025-08-25 01:48:41,456 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.48:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-08-25 01:48:41,456 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-08-25 01:48:41,456 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-25 01:48:41,465 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-25 01:48:41,471 - DEBUG - Do not have enough data to verify VN in agent 2025-08-25 01:48:41,471 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-25 01:48:41,478 - DEBUG - VRF ids for VN __public__: {} 2025-08-25 01:48:41,839 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-08-25 01:48:42,237 - DEBUG - Requesting: http://10.0.0.48:8082/domains 2025-08-25 01:48:42,266 - DEBUG - Requesting: http://10.0.0.48:8082/domain/a32a9a80-558e-4500-92e4-2d929868eed5 2025-08-25 01:48:42,288 - DEBUG - Requesting: http://10.0.0.48:8082/project/e7ebff81-059b-4396-840c-6ce2a40f909f 2025-08-25 01:48:42,452 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/d5a132f9-4473-4b77-95fd-93d8f851b554 2025-08-25 01:48:42,505 - DEBUG - Requesting: http://10.0.0.48:8082/floating-ip-pool/49ee91f6-75f0-44db-b5df-df4033b4e5c9 2025-08-25 01:48:42,520 - DEBUG - Requesting: http://10.0.0.48:8082/domains 2025-08-25 01:48:42,533 - DEBUG - Requesting: http://10.0.0.48:8082/domain/a32a9a80-558e-4500-92e4-2d929868eed5 2025-08-25 01:48:42,547 - DEBUG - Requesting: http://10.0.0.48:8082/project/e7ebff81-059b-4396-840c-6ce2a40f909f 2025-08-25 01:48:42,801 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/d5a132f9-4473-4b77-95fd-93d8f851b554 2025-08-25 01:48:42,820 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-08-25 01:48:42,820 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-08-25 01:48:42,842 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-08-25 01:48:42,863 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-08-25 01:48:42,863 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-08-25 01:48:42,867 - DEBUG - Ingress ctest-nginx-ingress-43761415 not present 2025-08-25 01:48:42,867 - INFO - Creating Ingress ctest-nginx-ingress-43761415 2025-08-25 01:48:42,876 - INFO - Created Ingress ctest-nginx-ingress-43761415 2025-08-25 01:48:42,880 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-43761415 2025-08-25 01:48:45,892 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-43761415 2025-08-25 01:48:48,898 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-43761415 2025-08-25 01:48:51,904 - DEBUG - For Ingress ctest-nginx-ingress-43761415, Cluster IP: 10.47.255.248, External IPs ['28.191.110.252'] 2025-08-25 01:48:51,905 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-43761415 2025-08-25 01:48:51,905 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_LoadbalancerDatabaseList?x=c431bc8d-d5bf-414e-beac-3acd5226a216 2025-08-25 01:48:51,916 - INFO - Ingress ctest-nginx-ingress-43761415 with uuid c431bc8d-d5bf-414e-beac-3acd5226a216 found in kube manager 2025-08-25 01:48:51,916 - INFO - Ingress ctest-nginx-ingress-43761415 verification passed 2025-08-25 01:48:51,922 - DEBUG - Pod ctest-busybox-pod-94118853 not present 2025-08-25 01:48:51,922 - INFO - Creating Pod ctest-busybox-pod-94118853 2025-08-25 01:48:51,961 - DEBUG - Pod : ctest-busybox-pod-94118853 UUID is 78bf3521-33cf-44f7-a2d0-11103819336a 2025-08-25 01:48:51,971 - DEBUG - Pod ctest-nginx-pod-96045388 not in running state.Currently in Pending 2025-08-25 01:48:56,980 - INFO - Pod ctest-nginx-pod-96045388 is in running state.Got IP 10.47.255.250 2025-08-25 01:48:57,175 - DEBUG - Pod ctest-nginx-pod-96045388 has vmi a1f391d8-8155-11f0-b050-02424bffd12a 2025-08-25 01:48:57,175 - INFO - Verified pod ctest-nginx-pod-96045388 in contrail-api 2025-08-25 01:48:57,175 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-08-25 01:48:57,196 - DEBUG - VMI a1f391d8-8155-11f0-b050-02424bffd12a is active in agent 10.0.0.48 2025-08-25 01:48:57,196 - INFO - Verified Pod ctest-nginx-pod-96045388 in agent 10.0.0.48 2025-08-25 01:48:57,197 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_VirtualMachineDatabaseList?x=20a8e29a-36cb-4b45-8596-4632ae5de0fb 2025-08-25 01:48:57,213 - INFO - Pod ctest-nginx-pod-96045388 with uuid 20a8e29a-36cb-4b45-8596-4632ae5de0fb found in kube manager 2025-08-25 01:48:57,213 - INFO - Pod ctest-nginx-pod-96045388 verification passed 2025-08-25 01:48:57,367 - DEBUG - [Pod ctest-nginx-pod-96045388] Cmd: echo ctest-nginx-pod-96045388 > /usr/share/nginx/html/index.html, Output: 2025-08-25 01:48:57,375 - INFO - Pod ctest-nginx-pod-65143093 is in running state.Got IP 10.47.255.249 2025-08-25 01:48:57,512 - DEBUG - Pod ctest-nginx-pod-65143093 has vmi a2dc8776-8155-11f0-b050-02424bffd12a 2025-08-25 01:48:57,512 - INFO - Verified pod ctest-nginx-pod-65143093 in contrail-api 2025-08-25 01:48:57,513 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-08-25 01:48:57,534 - DEBUG - VMI a2dc8776-8155-11f0-b050-02424bffd12a is active in agent 10.0.0.48 2025-08-25 01:48:57,535 - INFO - Verified Pod ctest-nginx-pod-65143093 in agent 10.0.0.48 2025-08-25 01:48:57,535 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_VirtualMachineDatabaseList?x=9bf56c28-b1f6-45da-9017-298e1672e102 2025-08-25 01:48:57,550 - INFO - Pod ctest-nginx-pod-65143093 with uuid 9bf56c28-b1f6-45da-9017-298e1672e102 found in kube manager 2025-08-25 01:48:57,550 - INFO - Pod ctest-nginx-pod-65143093 verification passed 2025-08-25 01:48:57,694 - DEBUG - [Pod ctest-nginx-pod-65143093] Cmd: echo ctest-nginx-pod-65143093 > /usr/share/nginx/html/index.html, Output: 2025-08-25 01:48:57,701 - DEBUG - Pod ctest-busybox-pod-94118853 not in running state.Currently in Pending 2025-08-25 01:49:02,710 - DEBUG - Pod ctest-busybox-pod-94118853 not in running state.Currently in Pending 2025-08-25 01:49:07,718 - INFO - Pod ctest-busybox-pod-94118853 is in running state.Got IP 10.47.255.247 2025-08-25 01:49:07,776 - DEBUG - Pod ctest-busybox-pod-94118853 has vmi a924d390-8155-11f0-b050-02424bffd12a 2025-08-25 01:49:07,777 - INFO - Verified pod ctest-busybox-pod-94118853 in contrail-api 2025-08-25 01:49:07,777 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-08-25 01:49:07,794 - DEBUG - VMI a924d390-8155-11f0-b050-02424bffd12a is active in agent 10.0.0.48 2025-08-25 01:49:07,795 - INFO - Verified Pod ctest-busybox-pod-94118853 in agent 10.0.0.48 2025-08-25 01:49:07,795 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_VirtualMachineDatabaseList?x=78bf3521-33cf-44f7-a2d0-11103819336a 2025-08-25 01:49:07,802 - INFO - Pod ctest-busybox-pod-94118853 with uuid 78bf3521-33cf-44f7-a2d0-11103819336a found in kube manager 2025-08-25 01:49:07,802 - INFO - Pod ctest-busybox-pod-94118853 verification passed 2025-08-25 01:49:07,922 - DEBUG - [Pod ctest-busybox-pod-94118853] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-65143093 - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-08-25 01:49:07,923 - DEBUG - [Pod Pod ctest-busybox-pod-94118853] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-08-25 01:49:07,923 - DEBUG - [Pod Pod ctest-busybox-pod-94118853] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-65143093 - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-08-25 01:49:08,027 - DEBUG - [Pod ctest-busybox-pod-94118853] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-96045388 2025-08-25 01:49:08,027 - DEBUG - [Pod Pod ctest-busybox-pod-94118853] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-08-25 01:49:08,027 - DEBUG - [Pod Pod ctest-busybox-pod-94118853] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-96045388 2025-08-25 01:49:08,027 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-96045388': 1, 'ctest-nginx-pod-65143093': 1} 2025-08-25 01:49:08,028 - WARNING - Nothing to delete parallely 2025-08-25 01:49:08,028 - INFO - Deleting pod default:ctest-busybox-pod-94118853 2025-08-25 01:49:08,040 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=78bf3521-33cf-44f7-a2d0-11103819336a 2025-08-25 01:49:08,052 - WARNING - Pod uuid 78bf3521-33cf-44f7-a2d0-11103819336a is still seen in agent 10.0.0.48 VM list 2025-08-25 01:49:13,052 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=78bf3521-33cf-44f7-a2d0-11103819336a 2025-08-25 01:49:13,060 - WARNING - Pod uuid 78bf3521-33cf-44f7-a2d0-11103819336a is still seen in agent 10.0.0.48 VM list 2025-08-25 01:49:18,061 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=78bf3521-33cf-44f7-a2d0-11103819336a 2025-08-25 01:49:18,068 - WARNING - Pod uuid 78bf3521-33cf-44f7-a2d0-11103819336a is still seen in agent 10.0.0.48 VM list 2025-08-25 01:49:23,069 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=78bf3521-33cf-44f7-a2d0-11103819336a 2025-08-25 01:49:23,076 - WARNING - Pod uuid 78bf3521-33cf-44f7-a2d0-11103819336a is still seen in agent 10.0.0.48 VM list 2025-08-25 01:49:28,077 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=78bf3521-33cf-44f7-a2d0-11103819336a 2025-08-25 01:49:28,084 - WARNING - Pod uuid 78bf3521-33cf-44f7-a2d0-11103819336a is still seen in agent 10.0.0.48 VM list 2025-08-25 01:49:33,084 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=78bf3521-33cf-44f7-a2d0-11103819336a 2025-08-25 01:49:33,093 - WARNING - Pod uuid 78bf3521-33cf-44f7-a2d0-11103819336a is still seen in agent 10.0.0.48 VM list 2025-08-25 01:49:38,094 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=78bf3521-33cf-44f7-a2d0-11103819336a 2025-08-25 01:49:38,102 - WARNING - Pod uuid 78bf3521-33cf-44f7-a2d0-11103819336a is still seen in agent 10.0.0.48 VM list 2025-08-25 01:49:43,102 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=78bf3521-33cf-44f7-a2d0-11103819336a 2025-08-25 01:49:43,113 - DEBUG - Pod 78bf3521-33cf-44f7-a2d0-11103819336a is not in agent 10.0.0.48 VM list 2025-08-25 01:49:43,113 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a924d390-8155-11f0-b050-02424bffd12a 2025-08-25 01:49:43,123 - DEBUG - VMI a924d390-8155-11f0-b050-02424bffd12a is removed from agent 10.0.0.48 2025-08-25 01:49:43,124 - INFO - Verified that pod ctest-busybox-pod-94118853 is removed in agent 2025-08-25 01:49:43,124 - INFO - Deleting Ingress : ctest-nginx-ingress-43761415 2025-08-25 01:49:43,135 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_LoadbalancerDatabaseList?x=c431bc8d-d5bf-414e-beac-3acd5226a216 2025-08-25 01:49:43,145 - ERROR - Ingress ctest-nginx-ingress-43761415 with uuid c431bc8d-d5bf-414e-beac-3acd5226a216 still found in kube manager 2025-08-25 01:49:44,146 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_LoadbalancerDatabaseList?x=c431bc8d-d5bf-414e-beac-3acd5226a216 2025-08-25 01:49:44,167 - ERROR - Ingress ctest-nginx-ingress-43761415 with uuid c431bc8d-d5bf-414e-beac-3acd5226a216 still found in kube manager 2025-08-25 01:49:45,168 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_LoadbalancerDatabaseList?x=c431bc8d-d5bf-414e-beac-3acd5226a216 2025-08-25 01:49:45,175 - INFO - Ingress ctest-nginx-ingress-43761415 with uuid c431bc8d-d5bf-414e-beac-3acd5226a216 deleted successfully from kube manager 2025-08-25 01:49:45,175 - INFO - Verifications on Ingress %s cleanup passed 2025-08-25 01:49:45,175 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-08-25 01:49:45,580 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-08-25 01:49:45,587 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-08-25 01:49:45,587 - INFO - Deleting VN __public__ 2025-08-25 01:49:46,092 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/2b5fca39-565d-4e12-a112-64fb89e0dad0 2025-08-25 01:49:46,107 - DEBUG - Response Code: 404 2025-08-25 01:49:46,107 - DEBUG - Requesting: http://10.0.0.48:8082/domains 2025-08-25 01:49:46,117 - DEBUG - Requesting: http://10.0.0.48:8082/domain/a32a9a80-558e-4500-92e4-2d929868eed5 2025-08-25 01:49:46,131 - DEBUG - Requesting: http://10.0.0.48:8082/project/e7ebff81-059b-4396-840c-6ce2a40f909f 2025-08-25 01:49:46,263 - INFO - Validated that VN __public__ is not found in API Server 2025-08-25 01:49:46,263 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-08-25 01:49:46,271 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-25 01:49:46,281 - DEBUG - VN __public__ is not present in Agent 10.0.0.48 2025-08-25 01:49:46,281 - INFO - Validated that VN __public__ is not in any agent 2025-08-25 01:49:46,281 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-25 01:49:46,293 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-25 01:49:46,306 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-08-25 01:49:46,314 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-08-25 01:49:46,315 - INFO - Deleting pod default:ctest-nginx-pod-65143093 2025-08-25 01:49:46,327 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=9bf56c28-b1f6-45da-9017-298e1672e102 2025-08-25 01:49:46,345 - WARNING - Pod uuid 9bf56c28-b1f6-45da-9017-298e1672e102 is still seen in agent 10.0.0.48 VM list 2025-08-25 01:49:51,346 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=9bf56c28-b1f6-45da-9017-298e1672e102 2025-08-25 01:49:51,352 - DEBUG - Pod 9bf56c28-b1f6-45da-9017-298e1672e102 is not in agent 10.0.0.48 VM list 2025-08-25 01:49:51,352 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a2dc8776-8155-11f0-b050-02424bffd12a 2025-08-25 01:49:51,358 - DEBUG - VMI a2dc8776-8155-11f0-b050-02424bffd12a is removed from agent 10.0.0.48 2025-08-25 01:49:51,358 - INFO - Verified that pod ctest-nginx-pod-65143093 is removed in agent 2025-08-25 01:49:51,358 - INFO - Deleting pod default:ctest-nginx-pod-96045388 2025-08-25 01:49:51,369 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=20a8e29a-36cb-4b45-8596-4632ae5de0fb 2025-08-25 01:49:51,377 - WARNING - Pod uuid 20a8e29a-36cb-4b45-8596-4632ae5de0fb is still seen in agent 10.0.0.48 VM list 2025-08-25 01:49:56,379 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_VmListReq?uuid=20a8e29a-36cb-4b45-8596-4632ae5de0fb 2025-08-25 01:49:56,386 - DEBUG - Pod 20a8e29a-36cb-4b45-8596-4632ae5de0fb is not in agent 10.0.0.48 VM list 2025-08-25 01:49:56,386 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a1f391d8-8155-11f0-b050-02424bffd12a 2025-08-25 01:49:56,393 - DEBUG - VMI a1f391d8-8155-11f0-b050-02424bffd12a is removed from agent 10.0.0.48 2025-08-25 01:49:56,393 - INFO - Verified that pod ctest-nginx-pod-96045388 is removed in agent 2025-08-25 01:49:56,393 - INFO - Deleting service : ctest-nginx-svc-02211479 2025-08-25 01:49:56,997 - DEBUG - Requesting: http://10.0.0.48:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-25 01:49:57,004 - DEBUG - No XMPP flaps were noticed during the test