2025-10-13 01:35:25,657 - DEBUG - [10.0.0.49]: Running cmd : hostname 2025-10-13 01:35:25,898 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-261-1 2025-10-13 01:35:25,898 - DEBUG - [10.0.0.49]: Running cmd : hostname -f 2025-10-13 01:35:26,024 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-261-1.cluster.local 2025-10-13 01:35:26,025 - DEBUG - [10.0.0.49]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-13 01:35:26,196 - DEBUG - Output : NAMES contrail_test_hvfofBBbx k8s_autoscaler_dns-autoscaler-595558c478-8hkmb_kube-system_abe8efa9-1838-48d9-b1f1-ef3995b01f9e_0 k8s_coredns_coredns-77f7cc69db-ccj82_kube-system_4766d3d0-8bb7-475b-8d87-038b021d4584_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-46xtq_kube-system_fda7b118-c4b2-41ce-b85b-893d1402e774_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-46xtq_kube-system_fda7b118-c4b2-41ce-b85b-893d1402e774_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-46xtq_kube-system_fda7b118-c4b2-41ce-b85b-893d1402e774_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-n79xf_kube-system_14f03593-3ef7-4b29-97cf-86b3c149f365_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-n7866_kube-system_94e6f239-3295-4330-bb45-d648fef0704f_0 k8s_opensdn-controller-control-named_opensdn-controller-control-n7866_kube-system_94e6f239-3295-4330-bb45-d648fef0704f_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-hms6j_kube-system_dc236941-3157-4b82-ad5a-735f2ac9ce6c_0 k8s_opensdn-analytics-collector_opensdn-analytics-hms6j_kube-system_dc236941-3157-4b82-ad5a-735f2ac9ce6c_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-gfvk6_kube-system_a08c1c0b-9d47-403d-a94a-d5689d463e65_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-gfvk6_kube-system_a08c1c0b-9d47-403d-a94a-d5689d463e65_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-46xtq_kube-system_fda7b118-c4b2-41ce-b85b-893d1402e774_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-lf4tz_kube-system_56559f17-c232-47db-b299-8a30c7b4256b_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-n79xf_kube-system_14f03593-3ef7-4b29-97cf-86b3c149f365_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-n7866_kube-system_94e6f239-3295-4330-bb45-d648fef0704f_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-p25j6_kube-system_84f7e887-4547-47c4-a237-d88c6c3bd543_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-p25j6_kube-system_84f7e887-4547-47c4-a237-d88c6c3bd543_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-ps4gn_kube-system_8961d896-7c3d-42a5-ae60-b1b05ff23626_0 k8s_opensdn-agent-nodemgr_opensdn-agent-b6gtk_kube-system_08fc44bd-7e98-48e5-a3f0-3c89574f1887_0 k8s_opensdn-analytics-api_opensdn-analytics-hms6j_kube-system_dc236941-3157-4b82-ad5a-735f2ac9ce6c_0 k8s_kafka_opensdn-analytics-alarm-gfvk6_kube-system_a08c1c0b-9d47-403d-a94a-d5689d463e65_0 k8s_opensdn-controller-config-api_opensdn-controller-config-46xtq_kube-system_fda7b118-c4b2-41ce-b85b-893d1402e774_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-lf4tz_kube-system_56559f17-c232-47db-b299-8a30c7b4256b_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-n79xf_kube-system_14f03593-3ef7-4b29-97cf-86b3c149f365_0 k8s_opensdn-controller-control_opensdn-controller-control-n7866_kube-system_94e6f239-3295-4330-bb45-d648fef0704f_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-p25j6_kube-system_84f7e887-4547-47c4-a237-d88c6c3bd543_0 k8s_opensdn-configdb_opensdn-configdb-ps4gn_kube-system_8961d896-7c3d-42a5-ae60-b1b05ff23626_0 k8s_redis_redis-nthdt_kube-system_12e15754-bc99-496e-92f4-9ba262cedc99_0 k8s_config-zookeeper_config-zookeeper-g6472_kube-system_d505a029-9bd3-454a-956c-801841af9f1e_0 k8s_rabbitmq_rabbitmq-4d9c4_kube-system_7692a62a-7608-4d36-a0b2-fd732aa4ab0c_0 k8s_opensdn-vrouter-agent_opensdn-agent-b6gtk_kube-system_08fc44bd-7e98-48e5-a3f0-3c89574f1887_0 k8s_opensdn-kube-manager_opensdn-kube-manager-kdwh9_kube-system_572e80ea-5a19-4110-b3cb-689734c4be7f_0 k8s_kube-proxy_kube-proxy-pwzb9_kube-system_5206510d-1fe1-4e49-95b0-67614253b594_0 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-261-1_kube-system_24e9e74de2d84f53a7423b822f6c0bc3_1 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-261-1_kube-system_3d8410aa467eaea0903446b6e469bde0_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-261-1_kube-system_e4cfaa0b339419797e8026fcf38aa807_0 2025-10-13 01:35:26,197 - DEBUG - [10.0.0.49]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-10-13 01:35:26,301 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.3/32 10.96.0.1/32 10.0.0.49/24 2025-10-13 01:35:26,301 - DEBUG - [10.0.0.49]: Running cmd : getent hosts 10.0.0.49 | head -n 1 | awk '{print $2}' 2025-10-13 01:35:26,415 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-261-1.cluster.local 2025-10-13 01:35:26,415 - DEBUG - [10.0.0.49]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-10-13 01:35:26,543 - DEBUG - Output : 10.0.0.49/24 2025-10-13 01:35:26,889 - INFO - Using existing project ['default-domain', 'k8s-default'](6f0bd1da-4af0-47a3-bfd3-738350bf6def) 2025-10-13 01:35:27,022 - INFO - Using existing project ['default-domain', 'k8s-default'](6f0bd1da-4af0-47a3-bfd3-738350bf6def) 2025-10-13 01:35:27,148 - INFO - ================================================================================ 2025-10-13 01:35:27,148 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-10-13 01:35:27,148 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-10-13 01:35:27,414 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-13 01:35:27,421 - DEBUG - Nothing to compare xmpp stats {'10.0.0.49': {'10.0.0.49': '3'}} with 2025-10-13 01:35:27,421 - INFO - Initial checks done. Running the testcase now 2025-10-13 01:35:27,421 - INFO - 2025-10-13 01:35:27,426 - INFO - Namespace default already exists 2025-10-13 01:35:27,426 - DEBUG - [10.0.0.49]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-kdwh9_kube-system_572e80ea-5a19-4110-b3cb-689734c4be7f_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-10-13 01:35:27,600 - DEBUG - Output : cluster_project={} 2025-10-13 01:35:27,670 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-10-13 01:35:27,670 - DEBUG - Project uuid in Contrail is 6f0bd1da-4af0-47a3-bfd3-738350bf6def 2025-10-13 01:35:27,671 - DEBUG - [10.0.0.49]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-kdwh9_kube-system_572e80ea-5a19-4110-b3cb-689734c4be7f_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-10-13 01:35:27,817 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-10-13 01:35:27,818 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_NamespaceDatabaseList?namespace_uuid=52753e38-2e7a-47d3-9f63-d4e5bc6d8368 2025-10-13 01:35:27,821 - INFO - Namespace default with uuid 52753e38-2e7a-47d3-9f63-d4e5bc6d8368 found in kube manager 2025-10-13 01:35:27,821 - INFO - Namespace default verification passed 2025-10-13 01:35:27,824 - DEBUG - Service ctest-nginx-svc-94266110 not present 2025-10-13 01:35:27,825 - INFO - Creating service ctest-nginx-svc-94266110 2025-10-13 01:35:27,841 - INFO - Created Service ctest-nginx-svc-94266110 2025-10-13 01:35:27,844 - DEBUG - Pod ctest-nginx-pod-98581588 not present 2025-10-13 01:35:27,847 - INFO - Creating Pod ctest-nginx-pod-98581588 2025-10-13 01:35:27,881 - DEBUG - Pod : ctest-nginx-pod-98581588 UUID is bbcb12b0-1423-4ffa-a011-475fe35e63ad 2025-10-13 01:35:27,886 - DEBUG - Pod ctest-nginx-pod-43240043 not present 2025-10-13 01:35:27,886 - INFO - Creating Pod ctest-nginx-pod-43240043 2025-10-13 01:35:27,915 - DEBUG - Pod : ctest-nginx-pod-43240043 UUID is fa9388ee-5e0e-46f0-b4bf-822f4236b8d1 2025-10-13 01:35:28,278 - INFO - Created VN __public__, UUID :97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:28,521 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-13 01:35:28,527 - DEBUG - Requesting: http://10.0.0.49:8082/domain/8c37ff7a-b720-46ef-8c73-5b1b1fdf4e79 2025-10-13 01:35:28,588 - DEBUG - Requesting: http://10.0.0.49:8082/project/6f0bd1da-4af0-47a3-bfd3-738350bf6def 2025-10-13 01:35:28,679 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:28,694 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:28,701 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/caaa4074-3c1a-4687-85ec-ead49608ed9e 2025-10-13 01:35:28,707 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/caaa4074-3c1a-4687-85ec-ead49608ed9e 2025-10-13 01:35:28,712 - DEBUG - Route Targets: [] 2025-10-13 01:35:28,712 - DEBUG - RT names not yet present for VN __public__ 2025-10-13 01:35:33,713 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-13 01:35:33,722 - DEBUG - Requesting: http://10.0.0.49:8082/domain/8c37ff7a-b720-46ef-8c73-5b1b1fdf4e79 2025-10-13 01:35:33,739 - DEBUG - Requesting: http://10.0.0.49:8082/project/6f0bd1da-4af0-47a3-bfd3-738350bf6def 2025-10-13 01:35:33,787 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:33,795 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:33,802 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/caaa4074-3c1a-4687-85ec-ead49608ed9e 2025-10-13 01:35:33,808 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/caaa4074-3c1a-4687-85ec-ead49608ed9e 2025-10-13 01:35:33,813 - DEBUG - Requesting: http://10.0.0.49:8082/route-target/3f71c97e-eff9-4a0c-aff2-ed3ccdf337f8 2025-10-13 01:35:33,819 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-13 01:35:33,819 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:33,826 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/caaa4074-3c1a-4687-85ec-ead49608ed9e 2025-10-13 01:35:33,836 - INFO - Verified VN network id 11 for VN 97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:33,836 - INFO - Verifications in API Server for VN __public__ passed 2025-10-13 01:35:33,836 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:33,843 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/caaa4074-3c1a-4687-85ec-ead49608ed9e 2025-10-13 01:35:33,851 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/caaa4074-3c1a-4687-85ec-ead49608ed9e 2025-10-13 01:35:33,857 - DEBUG - Requesting: http://10.0.0.49:8082/route-target/3f71c97e-eff9-4a0c-aff2-ed3ccdf337f8 2025-10-13 01:35:33,862 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-13 01:35:33,867 - DEBUG - Control-node 10.0.0.49 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '10895110649662754783', 'uuid-lslong': '12889100517485352796'}, 'enable': 'true', 'created': '2025-10-13T01:35:28', 'last-modified': '2025-10-13T01:35:28', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.460926'} 2025-10-13 01:35:33,867 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-13 01:35:33,874 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-13 01:35:33,874 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-10-13 01:35:33,875 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-10-13 01:35:33,875 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-13 01:35:33,880 - DEBUG - Requesting: http://10.0.0.49:8082/domain/8c37ff7a-b720-46ef-8c73-5b1b1fdf4e79 2025-10-13 01:35:33,886 - DEBUG - Requesting: http://10.0.0.49:8082/project/6f0bd1da-4af0-47a3-bfd3-738350bf6def 2025-10-13 01:35:33,913 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:33,922 - DEBUG - =>VN __public__ has no policy to be verified 2025-10-13 01:35:33,922 - DEBUG - Verifying the vn in opserver 2025-10-13 01:35:33,922 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.49 2025-10-13 01:35:33,923 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-networks 2025-10-13 01:35:33,926 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-10-13 01:35:33,926 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-10-13 01:35:33,926 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-13 01:35:33,931 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-13 01:35:33,937 - DEBUG - Do not have enough data to verify VN in agent 2025-10-13 01:35:33,937 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-13 01:35:33,941 - DEBUG - VRF ids for VN __public__: {} 2025-10-13 01:35:34,107 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-10-13 01:35:34,357 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-13 01:35:34,362 - DEBUG - Requesting: http://10.0.0.49:8082/domain/8c37ff7a-b720-46ef-8c73-5b1b1fdf4e79 2025-10-13 01:35:34,369 - DEBUG - Requesting: http://10.0.0.49:8082/project/6f0bd1da-4af0-47a3-bfd3-738350bf6def 2025-10-13 01:35:34,406 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:34,415 - DEBUG - Requesting: http://10.0.0.49:8082/floating-ip-pool/4b870a48-96b4-4fdc-b3a9-857c299d6325 2025-10-13 01:35:34,420 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-13 01:35:34,427 - DEBUG - Requesting: http://10.0.0.49:8082/domain/8c37ff7a-b720-46ef-8c73-5b1b1fdf4e79 2025-10-13 01:35:34,434 - DEBUG - Requesting: http://10.0.0.49:8082/project/6f0bd1da-4af0-47a3-bfd3-738350bf6def 2025-10-13 01:35:34,482 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/97333569-3db9-4fdf-b2df-48a19b04975c 2025-10-13 01:35:34,493 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-10-13 01:35:34,493 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-10-13 01:35:34,500 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-13 01:35:34,504 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-10-13 01:35:34,505 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-10-13 01:35:34,508 - DEBUG - Ingress ctest-nginx-ingress-44906999 not present 2025-10-13 01:35:34,509 - INFO - Creating Ingress ctest-nginx-ingress-44906999 2025-10-13 01:35:34,519 - INFO - Created Ingress ctest-nginx-ingress-44906999 2025-10-13 01:35:34,525 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-44906999 2025-10-13 01:35:37,531 - DEBUG - For Ingress ctest-nginx-ingress-44906999, Cluster IP: 10.47.255.248, External IPs ['159.211.178.188'] 2025-10-13 01:35:37,531 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-44906999 2025-10-13 01:35:37,532 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=66873a02-07e4-4489-bed1-a4647fc6b4e2 2025-10-13 01:35:37,541 - INFO - Ingress ctest-nginx-ingress-44906999 with uuid 66873a02-07e4-4489-bed1-a4647fc6b4e2 found in kube manager 2025-10-13 01:35:37,541 - INFO - Ingress ctest-nginx-ingress-44906999 verification passed 2025-10-13 01:35:37,545 - DEBUG - Pod ctest-busybox-pod-78813580 not present 2025-10-13 01:35:37,546 - INFO - Creating Pod ctest-busybox-pod-78813580 2025-10-13 01:35:37,574 - DEBUG - Pod : ctest-busybox-pod-78813580 UUID is 041395d1-c101-4e56-9fcb-eb5c0ff749f4 2025-10-13 01:35:37,600 - DEBUG - Pod ctest-nginx-pod-98581588 not in running state.Currently in Pending 2025-10-13 01:35:42,607 - INFO - Pod ctest-nginx-pod-98581588 is in running state.Got IP 10.47.255.250 2025-10-13 01:35:42,667 - DEBUG - Pod ctest-nginx-pod-98581588 has vmi e7517394-a7d4-11f0-ad40-0242e001ca1b 2025-10-13 01:35:42,667 - INFO - Verified pod ctest-nginx-pod-98581588 in contrail-api 2025-10-13 01:35:42,668 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-13 01:35:42,678 - DEBUG - VMI e7517394-a7d4-11f0-ad40-0242e001ca1b is active in agent 10.0.0.49 2025-10-13 01:35:42,678 - INFO - Verified Pod ctest-nginx-pod-98581588 in agent 10.0.0.49 2025-10-13 01:35:42,678 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=bbcb12b0-1423-4ffa-a011-475fe35e63ad 2025-10-13 01:35:42,682 - INFO - Pod ctest-nginx-pod-98581588 with uuid bbcb12b0-1423-4ffa-a011-475fe35e63ad found in kube manager 2025-10-13 01:35:42,682 - INFO - Pod ctest-nginx-pod-98581588 verification passed 2025-10-13 01:35:42,773 - DEBUG - [Pod ctest-nginx-pod-98581588] Cmd: echo ctest-nginx-pod-98581588 > /usr/share/nginx/html/index.html, Output: 2025-10-13 01:35:42,780 - INFO - Pod ctest-nginx-pod-43240043 is in running state.Got IP 10.47.255.249 2025-10-13 01:35:42,830 - DEBUG - Pod ctest-nginx-pod-43240043 has vmi e7ff1b8e-a7d4-11f0-ad40-0242e001ca1b 2025-10-13 01:35:42,830 - INFO - Verified pod ctest-nginx-pod-43240043 in contrail-api 2025-10-13 01:35:42,831 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-13 01:35:42,839 - DEBUG - VMI e7ff1b8e-a7d4-11f0-ad40-0242e001ca1b is active in agent 10.0.0.49 2025-10-13 01:35:42,839 - INFO - Verified Pod ctest-nginx-pod-43240043 in agent 10.0.0.49 2025-10-13 01:35:42,839 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=fa9388ee-5e0e-46f0-b4bf-822f4236b8d1 2025-10-13 01:35:42,843 - INFO - Pod ctest-nginx-pod-43240043 with uuid fa9388ee-5e0e-46f0-b4bf-822f4236b8d1 found in kube manager 2025-10-13 01:35:42,843 - INFO - Pod ctest-nginx-pod-43240043 verification passed 2025-10-13 01:35:42,937 - DEBUG - [Pod ctest-nginx-pod-43240043] Cmd: echo ctest-nginx-pod-43240043 > /usr/share/nginx/html/index.html, Output: 2025-10-13 01:35:42,945 - DEBUG - Pod ctest-busybox-pod-78813580 not in running state.Currently in Pending 2025-10-13 01:35:47,953 - DEBUG - Pod ctest-busybox-pod-78813580 not in running state.Currently in Pending 2025-10-13 01:35:52,961 - INFO - Pod ctest-busybox-pod-78813580 is in running state.Got IP 10.47.255.247 2025-10-13 01:35:53,012 - DEBUG - Pod ctest-busybox-pod-78813580 has vmi ed81949c-a7d4-11f0-ad40-0242e001ca1b 2025-10-13 01:35:53,012 - INFO - Verified pod ctest-busybox-pod-78813580 in contrail-api 2025-10-13 01:35:53,012 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-13 01:35:53,021 - DEBUG - VMI ed81949c-a7d4-11f0-ad40-0242e001ca1b is active in agent 10.0.0.49 2025-10-13 01:35:53,021 - INFO - Verified Pod ctest-busybox-pod-78813580 in agent 10.0.0.49 2025-10-13 01:35:53,021 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=041395d1-c101-4e56-9fcb-eb5c0ff749f4 2025-10-13 01:35:53,025 - INFO - Pod ctest-busybox-pod-78813580 with uuid 041395d1-c101-4e56-9fcb-eb5c0ff749f4 found in kube manager 2025-10-13 01:35:53,025 - INFO - Pod ctest-busybox-pod-78813580 verification passed 2025-10-13 01:35:53,090 - DEBUG - [Pod ctest-busybox-pod-78813580] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-98581588 2025-10-13 01:35:53,090 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-10-13 01:35:53,090 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-98581588 2025-10-13 01:35:53,159 - DEBUG - [Pod ctest-busybox-pod-78813580] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-98581588 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-13 01:35:53,159 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-10-13 01:35:53,159 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-98581588 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-13 01:35:53,237 - DEBUG - [Pod ctest-busybox-pod-78813580] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-98581588 - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-13 01:35:53,237 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-10-13 01:35:53,237 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-98581588 - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-13 01:35:53,301 - DEBUG - [Pod ctest-busybox-pod-78813580] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-98581588 2025-10-13 01:35:53,301 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-10-13 01:35:53,301 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-98581588 2025-10-13 01:35:53,371 - DEBUG - [Pod ctest-busybox-pod-78813580] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-98581588 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-13 01:35:53,371 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-10-13 01:35:53,371 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-98581588 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-13 01:35:53,445 - DEBUG - [Pod ctest-busybox-pod-78813580] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-43240043 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-13 01:35:53,446 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-10-13 01:35:53,446 - DEBUG - [Pod Pod ctest-busybox-pod-78813580] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-43240043 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-13 01:35:53,446 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-98581588': 5, 'ctest-nginx-pod-43240043': 1} 2025-10-13 01:35:53,446 - WARNING - Nothing to delete parallely 2025-10-13 01:35:53,446 - INFO - Deleting pod default:ctest-busybox-pod-78813580 2025-10-13 01:35:53,462 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=041395d1-c101-4e56-9fcb-eb5c0ff749f4 2025-10-13 01:35:53,469 - WARNING - Pod uuid 041395d1-c101-4e56-9fcb-eb5c0ff749f4 is still seen in agent 10.0.0.49 VM list 2025-10-13 01:35:58,469 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=041395d1-c101-4e56-9fcb-eb5c0ff749f4 2025-10-13 01:35:58,474 - WARNING - Pod uuid 041395d1-c101-4e56-9fcb-eb5c0ff749f4 is still seen in agent 10.0.0.49 VM list 2025-10-13 01:36:03,474 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=041395d1-c101-4e56-9fcb-eb5c0ff749f4 2025-10-13 01:36:03,479 - WARNING - Pod uuid 041395d1-c101-4e56-9fcb-eb5c0ff749f4 is still seen in agent 10.0.0.49 VM list 2025-10-13 01:36:08,479 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=041395d1-c101-4e56-9fcb-eb5c0ff749f4 2025-10-13 01:36:08,485 - WARNING - Pod uuid 041395d1-c101-4e56-9fcb-eb5c0ff749f4 is still seen in agent 10.0.0.49 VM list 2025-10-13 01:36:13,486 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=041395d1-c101-4e56-9fcb-eb5c0ff749f4 2025-10-13 01:36:13,492 - WARNING - Pod uuid 041395d1-c101-4e56-9fcb-eb5c0ff749f4 is still seen in agent 10.0.0.49 VM list 2025-10-13 01:36:18,492 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=041395d1-c101-4e56-9fcb-eb5c0ff749f4 2025-10-13 01:36:18,497 - WARNING - Pod uuid 041395d1-c101-4e56-9fcb-eb5c0ff749f4 is still seen in agent 10.0.0.49 VM list 2025-10-13 01:36:23,497 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=041395d1-c101-4e56-9fcb-eb5c0ff749f4 2025-10-13 01:36:23,502 - WARNING - Pod uuid 041395d1-c101-4e56-9fcb-eb5c0ff749f4 is still seen in agent 10.0.0.49 VM list 2025-10-13 01:36:28,503 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=041395d1-c101-4e56-9fcb-eb5c0ff749f4 2025-10-13 01:36:28,511 - DEBUG - Pod 041395d1-c101-4e56-9fcb-eb5c0ff749f4 is not in agent 10.0.0.49 VM list 2025-10-13 01:36:28,511 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ed81949c-a7d4-11f0-ad40-0242e001ca1b 2025-10-13 01:36:28,516 - DEBUG - VMI ed81949c-a7d4-11f0-ad40-0242e001ca1b is removed from agent 10.0.0.49 2025-10-13 01:36:28,516 - INFO - Verified that pod ctest-busybox-pod-78813580 is removed in agent 2025-10-13 01:36:28,516 - INFO - Deleting Ingress : ctest-nginx-ingress-44906999 2025-10-13 01:36:28,525 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=66873a02-07e4-4489-bed1-a4647fc6b4e2 2025-10-13 01:36:28,531 - ERROR - Ingress ctest-nginx-ingress-44906999 with uuid 66873a02-07e4-4489-bed1-a4647fc6b4e2 still found in kube manager 2025-10-13 01:36:29,532 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=66873a02-07e4-4489-bed1-a4647fc6b4e2 2025-10-13 01:36:29,536 - ERROR - Ingress ctest-nginx-ingress-44906999 with uuid 66873a02-07e4-4489-bed1-a4647fc6b4e2 still found in kube manager 2025-10-13 01:36:30,537 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=66873a02-07e4-4489-bed1-a4647fc6b4e2 2025-10-13 01:36:30,540 - INFO - Ingress ctest-nginx-ingress-44906999 with uuid 66873a02-07e4-4489-bed1-a4647fc6b4e2 deleted successfully from kube manager 2025-10-13 01:36:30,540 - INFO - Verifications on Ingress %s cleanup passed 2025-10-13 01:36:30,540 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-10-13 01:36:30,614 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-13 01:36:30,624 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-10-13 01:36:30,624 - INFO - Deleting VN __public__ 2025-10-13 01:36:30,690 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/caaa4074-3c1a-4687-85ec-ead49608ed9e 2025-10-13 01:36:30,694 - DEBUG - Response Code: 404 2025-10-13 01:36:30,694 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-13 01:36:30,698 - DEBUG - Requesting: http://10.0.0.49:8082/domain/8c37ff7a-b720-46ef-8c73-5b1b1fdf4e79 2025-10-13 01:36:30,708 - DEBUG - Requesting: http://10.0.0.49:8082/project/6f0bd1da-4af0-47a3-bfd3-738350bf6def 2025-10-13 01:36:30,732 - INFO - Validated that VN __public__ is not found in API Server 2025-10-13 01:36:30,732 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-10-13 01:36:30,739 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-13 01:36:30,743 - DEBUG - VN __public__ is not present in Agent 10.0.0.49 2025-10-13 01:36:30,743 - INFO - Validated that VN __public__ is not in any agent 2025-10-13 01:36:30,743 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-13 01:36:30,747 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-13 01:36:30,755 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-13 01:36:30,767 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-10-13 01:36:30,767 - INFO - Deleting pod default:ctest-nginx-pod-43240043 2025-10-13 01:36:30,782 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=fa9388ee-5e0e-46f0-b4bf-822f4236b8d1 2025-10-13 01:36:30,799 - WARNING - Pod uuid fa9388ee-5e0e-46f0-b4bf-822f4236b8d1 is still seen in agent 10.0.0.49 VM list 2025-10-13 01:36:35,800 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=fa9388ee-5e0e-46f0-b4bf-822f4236b8d1 2025-10-13 01:36:35,805 - DEBUG - Pod fa9388ee-5e0e-46f0-b4bf-822f4236b8d1 is not in agent 10.0.0.49 VM list 2025-10-13 01:36:35,805 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e7ff1b8e-a7d4-11f0-ad40-0242e001ca1b 2025-10-13 01:36:35,812 - DEBUG - VMI e7ff1b8e-a7d4-11f0-ad40-0242e001ca1b is removed from agent 10.0.0.49 2025-10-13 01:36:35,812 - INFO - Verified that pod ctest-nginx-pod-43240043 is removed in agent 2025-10-13 01:36:35,812 - INFO - Deleting pod default:ctest-nginx-pod-98581588 2025-10-13 01:36:35,838 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=bbcb12b0-1423-4ffa-a011-475fe35e63ad 2025-10-13 01:36:35,846 - WARNING - Pod uuid bbcb12b0-1423-4ffa-a011-475fe35e63ad is still seen in agent 10.0.0.49 VM list 2025-10-13 01:36:40,847 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=bbcb12b0-1423-4ffa-a011-475fe35e63ad 2025-10-13 01:36:40,852 - DEBUG - Pod bbcb12b0-1423-4ffa-a011-475fe35e63ad is not in agent 10.0.0.49 VM list 2025-10-13 01:36:40,852 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e7517394-a7d4-11f0-ad40-0242e001ca1b 2025-10-13 01:36:40,858 - DEBUG - VMI e7517394-a7d4-11f0-ad40-0242e001ca1b is removed from agent 10.0.0.49 2025-10-13 01:36:40,858 - INFO - Verified that pod ctest-nginx-pod-98581588 is removed in agent 2025-10-13 01:36:40,858 - INFO - Deleting service : ctest-nginx-svc-94266110 2025-10-13 01:36:41,387 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-13 01:36:41,392 - DEBUG - No XMPP flaps were noticed during the test