2026-11-09 01:49:18,624 - DEBUG - [10.0.0.50]: Running cmd : hostname 2026-11-09 01:49:18,916 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-439-1 2026-11-09 01:49:18,916 - DEBUG - [10.0.0.50]: Running cmd : hostname -f 2026-11-09 01:49:19,063 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-439-1.cluster.local 2026-11-09 01:49:19,063 - DEBUG - [10.0.0.50]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-11-09 01:49:19,281 - DEBUG - Output : NAMES contrail_test_Oe4lZEaur k8s_autoscaler_dns-autoscaler-595558c478-mm5lp_kube-system_cd314f15-db0f-460c-8b19-53d5a4580421_0 k8s_coredns_coredns-77f7cc69db-wkbv4_kube-system_cc54013b-32e2-407c-9c6b-f26873b2b7c6_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-6hhr7_kube-system_29d87771-51a3-4512-be46-2ec836f30c95_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-6hhr7_kube-system_29d87771-51a3-4512-be46-2ec836f30c95_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-6hhr7_kube-system_29d87771-51a3-4512-be46-2ec836f30c95_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-77p6c_kube-system_c1948078-c504-4214-bf7c-1da25cdab6a6_0 k8s_opensdn-controller-control-named_opensdn-controller-control-77p6c_kube-system_c1948078-c504-4214-bf7c-1da25cdab6a6_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-4jpl5_kube-system_f8b46411-e32f-42b0-9d00-4c9b8b4dd0e3_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-6hhr7_kube-system_29d87771-51a3-4512-be46-2ec836f30c95_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-ftqjr_kube-system_3213e860-3264-4900-8c62-460f81839ce3_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-ftqjr_kube-system_3213e860-3264-4900-8c62-460f81839ce3_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-5qxnp_kube-system_4d42c2a5-9a40-4347-a146-3f168bc0eb53_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-nvknw_kube-system_521dbcaf-0ead-4ddc-896b-15f7a7079dcd_0 k8s_opensdn-analytics-collector_opensdn-analytics-5qxnp_kube-system_4d42c2a5-9a40-4347-a146-3f168bc0eb53_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-xtlrl_kube-system_c48ec466-6130-4db2-b523-1f7316b74dff_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-xtlrl_kube-system_c48ec466-6130-4db2-b523-1f7316b74dff_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-77p6c_kube-system_c1948078-c504-4214-bf7c-1da25cdab6a6_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-4jpl5_kube-system_f8b46411-e32f-42b0-9d00-4c9b8b4dd0e3_0 k8s_opensdn-controller-config-api_opensdn-controller-config-6hhr7_kube-system_29d87771-51a3-4512-be46-2ec836f30c95_0 k8s_kafka_opensdn-analytics-alarm-ftqjr_kube-system_3213e860-3264-4900-8c62-460f81839ce3_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-nvknw_kube-system_521dbcaf-0ead-4ddc-896b-15f7a7079dcd_0 k8s_opensdn-analytics-api_opensdn-analytics-5qxnp_kube-system_4d42c2a5-9a40-4347-a146-3f168bc0eb53_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-xtlrl_kube-system_c48ec466-6130-4db2-b523-1f7316b74dff_0 k8s_opensdn-controller-control_opensdn-controller-control-77p6c_kube-system_c1948078-c504-4214-bf7c-1da25cdab6a6_0 k8s_rabbitmq_rabbitmq-p8p54_kube-system_394e084f-185a-4664-ba9e-7fdc65c58986_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-22tvl_kube-system_43737917-f243-49c6-9e10-ed5aff0f0433_0 k8s_opensdn-configdb_opensdn-configdb-22tvl_kube-system_43737917-f243-49c6-9e10-ed5aff0f0433_0 k8s_config-zookeeper_config-zookeeper-4wsnd_kube-system_02456457-551f-4fe3-898d-d443d9e2109d_0 k8s_redis_redis-j7lt5_kube-system_0436d7fd-0373-4f89-83a4-e820d720cf5d_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-4jpl5_kube-system_f8b46411-e32f-42b0-9d00-4c9b8b4dd0e3_0 k8s_opensdn-agent-nodemgr_opensdn-agent-5xj4f_kube-system_b6f22e48-a482-45c8-a1c3-ec81e108b8c3_0 k8s_opensdn-vrouter-agent_opensdn-agent-5xj4f_kube-system_b6f22e48-a482-45c8-a1c3-ec81e108b8c3_0 k8s_opensdn-kube-manager_opensdn-kube-manager-5jslg_kube-system_8aaa9a63-9f08-499b-91e3-2016f9e8f8b4_0 k8s_kube-proxy_kube-proxy-d29gg_kube-system_916e6dc9-844f-49ff-9a6d-b8585a9d036a_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-439-1_kube-system_f17acea876702426b3c99d10d77019ac_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-439-1_kube-system_6d98e0f565ce50f53b3baddfa33107a4_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-439-1_kube-system_6b47d977e9f31d61488f3b1b9f9ab6c6_0 2026-11-09 01:49:19,281 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-11-09 01:49:19,437 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.50/24 2026-11-09 01:49:19,437 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.0.0.50 | head -n 1 | awk '{print $2}' 2026-11-09 01:49:19,606 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-439-1.cluster.local 2026-11-09 01:49:19,606 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-11-09 01:49:19,773 - DEBUG - Output : 10.0.0.50/24 2026-11-09 01:49:20,234 - INFO - Using existing project ['default-domain', 'k8s-default'](a7cacd13-1246-4bad-a981-61ef7c6d616b) 2026-11-09 01:49:20,439 - INFO - Using existing project ['default-domain', 'k8s-default'](a7cacd13-1246-4bad-a981-61ef7c6d616b) 2026-11-09 01:49:20,601 - INFO - ================================================================================ 2026-11-09 01:49:20,601 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-11-09 01:49:20,601 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-11-09 01:49:20,981 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq? 2026-11-09 01:49:20,988 - DEBUG - Nothing to compare xmpp stats {'10.0.0.50': {'10.0.0.50': '2'}} with 2026-11-09 01:49:20,988 - INFO - Initial checks done. Running the testcase now 2026-11-09 01:49:20,988 - INFO - 2026-11-09 01:49:20,993 - INFO - Namespace default already exists 2026-11-09 01:49:20,993 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-5jslg_kube-system_8aaa9a63-9f08-499b-91e3-2016f9e8f8b4_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-11-09 01:49:21,198 - DEBUG - Output : cluster_project={} 2026-11-09 01:49:21,272 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-11-09 01:49:21,272 - DEBUG - Project uuid in Contrail is a7cacd13-1246-4bad-a981-61ef7c6d616b 2026-11-09 01:49:21,272 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-5jslg_kube-system_8aaa9a63-9f08-499b-91e3-2016f9e8f8b4_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-11-09 01:49:21,472 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-11-09 01:49:21,473 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_NamespaceDatabaseList?namespace_uuid=70b93a49-5a74-454d-a2f7-cbfd9b1fa648 2026-11-09 01:49:21,480 - INFO - Namespace default with uuid 70b93a49-5a74-454d-a2f7-cbfd9b1fa648 found in kube manager 2026-11-09 01:49:21,480 - INFO - Namespace default verification passed 2026-11-09 01:49:21,484 - DEBUG - Service ctest-nginx-svc-97249811 not present 2026-11-09 01:49:21,485 - INFO - Creating service ctest-nginx-svc-97249811 2026-11-09 01:49:21,500 - INFO - Created Service ctest-nginx-svc-97249811 2026-11-09 01:49:21,511 - DEBUG - Pod ctest-nginx-pod-03750005 not present 2026-11-09 01:49:21,513 - INFO - Creating Pod ctest-nginx-pod-03750005 2026-11-09 01:49:21,541 - DEBUG - Pod : ctest-nginx-pod-03750005 UUID is 4cef19ad-cd4f-42d5-b83f-5ad3f6209893 2026-11-09 01:49:21,551 - DEBUG - Pod ctest-nginx-pod-20628837 not present 2026-11-09 01:49:21,552 - INFO - Creating Pod ctest-nginx-pod-20628837 2026-11-09 01:49:21,580 - DEBUG - Pod : ctest-nginx-pod-20628837 UUID is c4d79e8c-1b96-4b37-ba1e-9f039cb416b7 2026-11-09 01:49:22,014 - INFO - Created VN __public__, UUID :43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:22,315 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-11-09 01:49:22,334 - DEBUG - Requesting: http://10.0.0.50:8082/domain/131fc2b5-a19d-4881-a37c-c2f35233f586 2026-11-09 01:49:22,425 - DEBUG - Requesting: http://10.0.0.50:8082/project/a7cacd13-1246-4bad-a981-61ef7c6d616b 2026-11-09 01:49:22,553 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:22,580 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:22,595 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a41ba14b-70e0-4f20-b407-7976037fb258 2026-11-09 01:49:22,603 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a41ba14b-70e0-4f20-b407-7976037fb258 2026-11-09 01:49:22,617 - DEBUG - Route Targets: [] 2026-11-09 01:49:22,617 - DEBUG - RT names not yet present for VN __public__ 2026-11-09 01:49:27,617 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-11-09 01:49:27,630 - DEBUG - Requesting: http://10.0.0.50:8082/domain/131fc2b5-a19d-4881-a37c-c2f35233f586 2026-11-09 01:49:27,639 - DEBUG - Requesting: http://10.0.0.50:8082/project/a7cacd13-1246-4bad-a981-61ef7c6d616b 2026-11-09 01:49:27,689 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:27,703 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:27,712 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a41ba14b-70e0-4f20-b407-7976037fb258 2026-11-09 01:49:27,724 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a41ba14b-70e0-4f20-b407-7976037fb258 2026-11-09 01:49:27,735 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/e8baa368-dea1-43e8-aa82-83ad0c8ae9e0 2026-11-09 01:49:27,742 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-11-09 01:49:27,742 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:27,751 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a41ba14b-70e0-4f20-b407-7976037fb258 2026-11-09 01:49:27,764 - INFO - Verified VN network id 11 for VN 43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:27,765 - INFO - Verifications in API Server for VN __public__ passed 2026-11-09 01:49:27,765 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:27,773 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a41ba14b-70e0-4f20-b407-7976037fb258 2026-11-09 01:49:27,778 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a41ba14b-70e0-4f20-b407-7976037fb258 2026-11-09 01:49:27,784 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/e8baa368-dea1-43e8-aa82-83ad0c8ae9e0 2026-11-09 01:49:27,790 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-11-09 01:49:27,797 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4893995713868350478', 'uuid-lslong': '9969803186070818003'}, 'enable': 'true', 'created': '2026-11-09T01:49:21', 'last-modified': '2026-11-09T01:49:22', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.604683'} 2026-11-09 01:49:27,797 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-11-09 01:49:27,806 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-11-09 01:49:27,806 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-11-09 01:49:27,807 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-11-09 01:49:27,807 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-11-09 01:49:27,813 - DEBUG - Requesting: http://10.0.0.50:8082/domain/131fc2b5-a19d-4881-a37c-c2f35233f586 2026-11-09 01:49:27,820 - DEBUG - Requesting: http://10.0.0.50:8082/project/a7cacd13-1246-4bad-a981-61ef7c6d616b 2026-11-09 01:49:27,852 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:27,862 - DEBUG - =>VN __public__ has no policy to be verified 2026-11-09 01:49:27,862 - DEBUG - Verifying the vn in opserver 2026-11-09 01:49:27,862 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.50 2026-11-09 01:49:27,862 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-11-09 01:49:27,867 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-11-09 01:49:27,867 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-11-09 01:49:27,867 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-11-09 01:49:27,874 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-11-09 01:49:27,879 - DEBUG - Do not have enough data to verify VN in agent 2026-11-09 01:49:27,879 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-11-09 01:49:27,884 - DEBUG - VRF ids for VN __public__: {} 2026-11-09 01:49:28,052 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-11-09 01:49:28,268 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-11-09 01:49:28,274 - DEBUG - Requesting: http://10.0.0.50:8082/domain/131fc2b5-a19d-4881-a37c-c2f35233f586 2026-11-09 01:49:28,285 - DEBUG - Requesting: http://10.0.0.50:8082/project/a7cacd13-1246-4bad-a981-61ef7c6d616b 2026-11-09 01:49:28,331 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:28,348 - DEBUG - Requesting: http://10.0.0.50:8082/floating-ip-pool/c971ddc6-077a-4896-adb1-4855aa17ceec 2026-11-09 01:49:28,356 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-11-09 01:49:28,366 - DEBUG - Requesting: http://10.0.0.50:8082/domain/131fc2b5-a19d-4881-a37c-c2f35233f586 2026-11-09 01:49:28,374 - DEBUG - Requesting: http://10.0.0.50:8082/project/a7cacd13-1246-4bad-a981-61ef7c6d616b 2026-11-09 01:49:28,410 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/43eaf72c-1968-440e-8a5b-db2cea4f0cd3 2026-11-09 01:49:28,422 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-11-09 01:49:28,422 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-11-09 01:49:28,432 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-11-09 01:49:28,441 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-11-09 01:49:28,441 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-11-09 01:49:28,445 - DEBUG - Ingress ctest-nginx-ingress-48831515 not present 2026-11-09 01:49:28,446 - INFO - Creating Ingress ctest-nginx-ingress-48831515 2026-11-09 01:49:28,453 - INFO - Created Ingress ctest-nginx-ingress-48831515 2026-11-09 01:49:28,460 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-48831515 2026-11-09 01:49:31,466 - DEBUG - For Ingress ctest-nginx-ingress-48831515, Cluster IP: 10.47.255.248, External IPs ['39.22.153.124'] 2026-11-09 01:49:31,466 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-48831515 2026-11-09 01:49:31,467 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=81bf4f6b-50aa-4c1d-bd50-202445444862 2026-11-09 01:49:31,472 - INFO - Ingress ctest-nginx-ingress-48831515 with uuid 81bf4f6b-50aa-4c1d-bd50-202445444862 found in kube manager 2026-11-09 01:49:31,472 - INFO - Ingress ctest-nginx-ingress-48831515 verification passed 2026-11-09 01:49:31,485 - DEBUG - Pod ctest-busybox-pod-59155369 not present 2026-11-09 01:49:31,486 - INFO - Creating Pod ctest-busybox-pod-59155369 2026-11-09 01:49:31,516 - DEBUG - Pod : ctest-busybox-pod-59155369 UUID is 002d5c2a-6fd6-404b-a1dc-92ecb44bb751 2026-11-09 01:49:31,526 - DEBUG - Pod ctest-nginx-pod-03750005 not in running state.Currently in Pending 2026-11-09 01:49:36,537 - DEBUG - Pod ctest-nginx-pod-03750005 not in running state.Currently in Pending 2026-11-09 01:49:41,546 - DEBUG - Pod ctest-nginx-pod-03750005 not in running state.Currently in Pending 2026-11-09 01:49:46,556 - INFO - Pod ctest-nginx-pod-03750005 is in running state.Got IP 10.47.255.250 2026-11-09 01:49:46,611 - DEBUG - Pod ctest-nginx-pod-03750005 has vmi ca76d080-dbdf-11f1-8072-02426f75ea1b 2026-11-09 01:49:46,612 - INFO - Verified pod ctest-nginx-pod-03750005 in contrail-api 2026-11-09 01:49:46,612 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-11-09 01:49:46,631 - DEBUG - VMI ca76d080-dbdf-11f1-8072-02426f75ea1b is active in agent 10.0.0.50 2026-11-09 01:49:46,635 - INFO - Verified Pod ctest-nginx-pod-03750005 in agent 10.0.0.50 2026-11-09 01:49:46,635 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=4cef19ad-cd4f-42d5-b83f-5ad3f6209893 2026-11-09 01:49:46,641 - INFO - Pod ctest-nginx-pod-03750005 with uuid 4cef19ad-cd4f-42d5-b83f-5ad3f6209893 found in kube manager 2026-11-09 01:49:46,641 - INFO - Pod ctest-nginx-pod-03750005 verification passed 2026-11-09 01:49:46,749 - DEBUG - [Pod ctest-nginx-pod-03750005] Cmd: echo ctest-nginx-pod-03750005 > /usr/share/nginx/html/index.html, Output: 2026-11-09 01:49:46,758 - INFO - Pod ctest-nginx-pod-20628837 is in running state.Got IP 10.47.255.249 2026-11-09 01:49:46,810 - DEBUG - Pod ctest-nginx-pod-20628837 has vmi cb3d1cc2-dbdf-11f1-8072-02426f75ea1b 2026-11-09 01:49:46,810 - INFO - Verified pod ctest-nginx-pod-20628837 in contrail-api 2026-11-09 01:49:46,811 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-11-09 01:49:46,822 - DEBUG - VMI cb3d1cc2-dbdf-11f1-8072-02426f75ea1b is active in agent 10.0.0.50 2026-11-09 01:49:46,822 - INFO - Verified Pod ctest-nginx-pod-20628837 in agent 10.0.0.50 2026-11-09 01:49:46,822 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=c4d79e8c-1b96-4b37-ba1e-9f039cb416b7 2026-11-09 01:49:46,828 - INFO - Pod ctest-nginx-pod-20628837 with uuid c4d79e8c-1b96-4b37-ba1e-9f039cb416b7 found in kube manager 2026-11-09 01:49:46,828 - INFO - Pod ctest-nginx-pod-20628837 verification passed 2026-11-09 01:49:46,934 - DEBUG - [Pod ctest-nginx-pod-20628837] Cmd: echo ctest-nginx-pod-20628837 > /usr/share/nginx/html/index.html, Output: 2026-11-09 01:49:46,942 - INFO - Pod ctest-busybox-pod-59155369 is in running state.Got IP 10.47.255.247 2026-11-09 01:49:46,998 - DEBUG - Pod ctest-busybox-pod-59155369 has vmi d017a5fa-dbdf-11f1-8072-02426f75ea1b 2026-11-09 01:49:46,999 - INFO - Verified pod ctest-busybox-pod-59155369 in contrail-api 2026-11-09 01:49:46,999 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-11-09 01:49:47,009 - DEBUG - VMI d017a5fa-dbdf-11f1-8072-02426f75ea1b is active in agent 10.0.0.50 2026-11-09 01:49:47,009 - INFO - Verified Pod ctest-busybox-pod-59155369 in agent 10.0.0.50 2026-11-09 01:49:47,009 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=002d5c2a-6fd6-404b-a1dc-92ecb44bb751 2026-11-09 01:49:47,013 - INFO - Pod ctest-busybox-pod-59155369 with uuid 002d5c2a-6fd6-404b-a1dc-92ecb44bb751 found in kube manager 2026-11-09 01:49:47,014 - INFO - Pod ctest-busybox-pod-59155369 verification passed 2026-11-09 01:49:47,100 - DEBUG - [Pod ctest-busybox-pod-59155369] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-03750005 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-11-09 01:49:47,100 - DEBUG - [Pod Pod ctest-busybox-pod-59155369] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-11-09 01:49:47,101 - DEBUG - [Pod Pod ctest-busybox-pod-59155369] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-03750005 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-11-09 01:49:47,187 - DEBUG - [Pod ctest-busybox-pod-59155369] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-03750005 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-11-09 01:49:47,187 - DEBUG - [Pod Pod ctest-busybox-pod-59155369] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-11-09 01:49:47,187 - DEBUG - [Pod Pod ctest-busybox-pod-59155369] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-03750005 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-11-09 01:49:47,266 - DEBUG - [Pod ctest-busybox-pod-59155369] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-03750005 2026-11-09 01:49:47,266 - DEBUG - [Pod Pod ctest-busybox-pod-59155369] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-11-09 01:49:47,266 - DEBUG - [Pod Pod ctest-busybox-pod-59155369] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-03750005 2026-11-09 01:49:47,352 - DEBUG - [Pod ctest-busybox-pod-59155369] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-03750005 Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-11-09 01:49:47,352 - DEBUG - [Pod Pod ctest-busybox-pod-59155369] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-11-09 01:49:47,352 - DEBUG - [Pod Pod ctest-busybox-pod-59155369] Cmd output: ctest-nginx-pod-03750005 Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-11-09 01:49:47,436 - DEBUG - [Pod ctest-busybox-pod-59155369] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-20628837 Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-11-09 01:49:47,437 - DEBUG - [Pod Pod ctest-busybox-pod-59155369] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-11-09 01:49:47,437 - DEBUG - [Pod Pod ctest-busybox-pod-59155369] Cmd output: ctest-nginx-pod-20628837 Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-11-09 01:49:47,437 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-03750005': 4, 'ctest-nginx-pod-20628837': 1} 2026-11-09 01:49:47,437 - WARNING - Nothing to delete parallely 2026-11-09 01:49:47,437 - INFO - Deleting pod default:ctest-busybox-pod-59155369 2026-11-09 01:49:47,455 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=002d5c2a-6fd6-404b-a1dc-92ecb44bb751 2026-11-09 01:49:47,465 - WARNING - Pod uuid 002d5c2a-6fd6-404b-a1dc-92ecb44bb751 is still seen in agent 10.0.0.50 VM list 2026-11-09 01:49:52,466 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=002d5c2a-6fd6-404b-a1dc-92ecb44bb751 2026-11-09 01:49:52,475 - WARNING - Pod uuid 002d5c2a-6fd6-404b-a1dc-92ecb44bb751 is still seen in agent 10.0.0.50 VM list 2026-11-09 01:49:57,476 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=002d5c2a-6fd6-404b-a1dc-92ecb44bb751 2026-11-09 01:49:57,485 - WARNING - Pod uuid 002d5c2a-6fd6-404b-a1dc-92ecb44bb751 is still seen in agent 10.0.0.50 VM list 2026-11-09 01:50:02,485 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=002d5c2a-6fd6-404b-a1dc-92ecb44bb751 2026-11-09 01:50:02,493 - WARNING - Pod uuid 002d5c2a-6fd6-404b-a1dc-92ecb44bb751 is still seen in agent 10.0.0.50 VM list 2026-11-09 01:50:07,493 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=002d5c2a-6fd6-404b-a1dc-92ecb44bb751 2026-11-09 01:50:07,500 - WARNING - Pod uuid 002d5c2a-6fd6-404b-a1dc-92ecb44bb751 is still seen in agent 10.0.0.50 VM list 2026-11-09 01:50:12,501 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=002d5c2a-6fd6-404b-a1dc-92ecb44bb751 2026-11-09 01:50:12,508 - WARNING - Pod uuid 002d5c2a-6fd6-404b-a1dc-92ecb44bb751 is still seen in agent 10.0.0.50 VM list 2026-11-09 01:50:17,509 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=002d5c2a-6fd6-404b-a1dc-92ecb44bb751 2026-11-09 01:50:17,517 - WARNING - Pod uuid 002d5c2a-6fd6-404b-a1dc-92ecb44bb751 is still seen in agent 10.0.0.50 VM list 2026-11-09 01:50:22,518 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=002d5c2a-6fd6-404b-a1dc-92ecb44bb751 2026-11-09 01:50:22,524 - DEBUG - Pod 002d5c2a-6fd6-404b-a1dc-92ecb44bb751 is not in agent 10.0.0.50 VM list 2026-11-09 01:50:22,524 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d017a5fa-dbdf-11f1-8072-02426f75ea1b 2026-11-09 01:50:22,534 - DEBUG - VMI d017a5fa-dbdf-11f1-8072-02426f75ea1b is removed from agent 10.0.0.50 2026-11-09 01:50:22,534 - INFO - Verified that pod ctest-busybox-pod-59155369 is removed in agent 2026-11-09 01:50:22,535 - INFO - Deleting Ingress : ctest-nginx-ingress-48831515 2026-11-09 01:50:22,545 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=81bf4f6b-50aa-4c1d-bd50-202445444862 2026-11-09 01:50:22,551 - ERROR - Ingress ctest-nginx-ingress-48831515 with uuid 81bf4f6b-50aa-4c1d-bd50-202445444862 still found in kube manager 2026-11-09 01:50:23,552 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=81bf4f6b-50aa-4c1d-bd50-202445444862 2026-11-09 01:50:23,558 - ERROR - Ingress ctest-nginx-ingress-48831515 with uuid 81bf4f6b-50aa-4c1d-bd50-202445444862 still found in kube manager 2026-11-09 01:50:24,559 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=81bf4f6b-50aa-4c1d-bd50-202445444862 2026-11-09 01:50:24,563 - INFO - Ingress ctest-nginx-ingress-48831515 with uuid 81bf4f6b-50aa-4c1d-bd50-202445444862 deleted successfully from kube manager 2026-11-09 01:50:24,563 - INFO - Verifications on Ingress %s cleanup passed 2026-11-09 01:50:24,563 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-11-09 01:50:24,763 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-11-09 01:50:24,772 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-11-09 01:50:24,772 - INFO - Deleting VN __public__ 2026-11-09 01:50:25,089 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/a41ba14b-70e0-4f20-b407-7976037fb258 2026-11-09 01:50:25,101 - DEBUG - Response Code: 404 2026-11-09 01:50:25,101 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-11-09 01:50:25,112 - DEBUG - Requesting: http://10.0.0.50:8082/domain/131fc2b5-a19d-4881-a37c-c2f35233f586 2026-11-09 01:50:25,123 - DEBUG - Requesting: http://10.0.0.50:8082/project/a7cacd13-1246-4bad-a981-61ef7c6d616b 2026-11-09 01:50:25,186 - INFO - Validated that VN __public__ is not found in API Server 2026-11-09 01:50:25,186 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-11-09 01:50:25,195 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-11-09 01:50:25,203 - DEBUG - VN __public__ is not present in Agent 10.0.0.50 2026-11-09 01:50:25,203 - INFO - Validated that VN __public__ is not in any agent 2026-11-09 01:50:25,204 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-11-09 01:50:25,223 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-11-09 01:50:25,232 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-11-09 01:50:25,239 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-11-09 01:50:25,240 - INFO - Deleting pod default:ctest-nginx-pod-20628837 2026-11-09 01:50:25,257 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=c4d79e8c-1b96-4b37-ba1e-9f039cb416b7 2026-11-09 01:50:25,277 - WARNING - Pod uuid c4d79e8c-1b96-4b37-ba1e-9f039cb416b7 is still seen in agent 10.0.0.50 VM list 2026-11-09 01:50:30,278 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=c4d79e8c-1b96-4b37-ba1e-9f039cb416b7 2026-11-09 01:50:30,283 - DEBUG - Pod c4d79e8c-1b96-4b37-ba1e-9f039cb416b7 is not in agent 10.0.0.50 VM list 2026-11-09 01:50:30,283 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:cb3d1cc2-dbdf-11f1-8072-02426f75ea1b 2026-11-09 01:50:30,288 - DEBUG - VMI cb3d1cc2-dbdf-11f1-8072-02426f75ea1b is removed from agent 10.0.0.50 2026-11-09 01:50:30,288 - INFO - Verified that pod ctest-nginx-pod-20628837 is removed in agent 2026-11-09 01:50:30,288 - INFO - Deleting pod default:ctest-nginx-pod-03750005 2026-11-09 01:50:30,304 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=4cef19ad-cd4f-42d5-b83f-5ad3f6209893 2026-11-09 01:50:30,316 - WARNING - Pod uuid 4cef19ad-cd4f-42d5-b83f-5ad3f6209893 is still seen in agent 10.0.0.50 VM list 2026-11-09 01:50:35,317 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=4cef19ad-cd4f-42d5-b83f-5ad3f6209893 2026-11-09 01:50:35,324 - DEBUG - Pod 4cef19ad-cd4f-42d5-b83f-5ad3f6209893 is not in agent 10.0.0.50 VM list 2026-11-09 01:50:35,324 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ca76d080-dbdf-11f1-8072-02426f75ea1b 2026-11-09 01:50:35,330 - DEBUG - VMI ca76d080-dbdf-11f1-8072-02426f75ea1b is removed from agent 10.0.0.50 2026-11-09 01:50:35,330 - INFO - Verified that pod ctest-nginx-pod-03750005 is removed in agent 2026-11-09 01:50:35,331 - INFO - Deleting service : ctest-nginx-svc-97249811 2026-11-09 01:50:36,015 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq? 2026-11-09 01:50:36,022 - DEBUG - No XMPP flaps were noticed during the test