2025-10-09 01:38:00,255 - DEBUG - [10.0.0.49]: Running cmd : hostname 2025-10-09 01:38:00,540 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-260-1 2025-10-09 01:38:00,540 - DEBUG - [10.0.0.49]: Running cmd : hostname -f 2025-10-09 01:38:00,665 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local 2025-10-09 01:38:00,665 - DEBUG - [10.0.0.49]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-09 01:38:00,839 - DEBUG - Output : NAMES contrail_test_udgkDaE4I k8s_autoscaler_dns-autoscaler-595558c478-f7bpc_kube-system_38fb3b19-17a6-43d4-8cfa-88a1ef402274_0 k8s_coredns_coredns-77f7cc69db-qxqqf_kube-system_091dfb53-d444-4a35-ad12-682615939c12_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-6kx75_kube-system_819a3399-db24-4d2e-88ad-4dfef390e428_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-6kx75_kube-system_819a3399-db24-4d2e-88ad-4dfef390e428_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-qfj54_kube-system_1839b40c-7963-4bae-ae07-cc31808bd362_0 k8s_opensdn-controller-control-named_opensdn-controller-control-qfj54_kube-system_1839b40c-7963-4bae-ae07-cc31808bd362_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-hjxpk_kube-system_ce0aaf59-4253-46f1-8ae0-dcc126253f4b_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-kd258_kube-system_92295879-d044-44c1-899b-ab610172beb2_1 k8s_opensdn-controller-config-schema_opensdn-controller-config-6kx75_kube-system_819a3399-db24-4d2e-88ad-4dfef390e428_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-pznlq_kube-system_c5223d9b-0cfa-4ded-bb29-f1f8a1effc87_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-pznlq_kube-system_c5223d9b-0cfa-4ded-bb29-f1f8a1effc87_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-877gz_kube-system_677d435c-3a49-4261-a3b7-8a2be775c477_0 k8s_opensdn-analytics-collector_opensdn-analytics-877gz_kube-system_677d435c-3a49-4261-a3b7-8a2be775c477_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-qfj54_kube-system_1839b40c-7963-4bae-ae07-cc31808bd362_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-wkmw7_kube-system_f89f4721-bffb-4d71-835b-44f0ff60d6b4_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-hjxpk_kube-system_ce0aaf59-4253-46f1-8ae0-dcc126253f4b_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-kd258_kube-system_92295879-d044-44c1-899b-ab610172beb2_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-6kx75_kube-system_819a3399-db24-4d2e-88ad-4dfef390e428_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-9cbwj_kube-system_1f21e8af-4164-4b2f-859c-9c782b9efeb5_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-pznlq_kube-system_c5223d9b-0cfa-4ded-bb29-f1f8a1effc87_0 k8s_opensdn-analytics-api_opensdn-analytics-877gz_kube-system_677d435c-3a49-4261-a3b7-8a2be775c477_0 k8s_opensdn-controller-control_opensdn-controller-control-qfj54_kube-system_1839b40c-7963-4bae-ae07-cc31808bd362_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-wkmw7_kube-system_f89f4721-bffb-4d71-835b-44f0ff60d6b4_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-hjxpk_kube-system_ce0aaf59-4253-46f1-8ae0-dcc126253f4b_0 k8s_kafka_opensdn-analytics-alarm-kd258_kube-system_92295879-d044-44c1-899b-ab610172beb2_0 k8s_opensdn-agent-nodemgr_opensdn-agent-5trvk_kube-system_0c051767-56cd-4a01-8f15-2f4665767c0d_0 k8s_opensdn-controller-config-api_opensdn-controller-config-6kx75_kube-system_819a3399-db24-4d2e-88ad-4dfef390e428_0 k8s_config-zookeeper_config-zookeeper-6sjd8_kube-system_0d072853-0542-4ca2-adb6-43cb4cf1e363_0 k8s_redis_redis-76ls6_kube-system_a826acbc-9752-4530-abc7-338abb949002_0 k8s_opensdn-configdb_opensdn-configdb-9cbwj_kube-system_1f21e8af-4164-4b2f-859c-9c782b9efeb5_0 k8s_rabbitmq_rabbitmq-bm9rl_kube-system_ca2660cd-ae1d-4707-bdcf-cad877ae0c3b_0 k8s_opensdn-vrouter-agent_opensdn-agent-5trvk_kube-system_0c051767-56cd-4a01-8f15-2f4665767c0d_0 k8s_opensdn-kube-manager_opensdn-kube-manager-9csct_kube-system_03af9417-4b03-4ab9-89dc-818d0acc99bd_0 k8s_kube-proxy_kube-proxy-27fdf_kube-system_a6855a36-a24e-4485-8787-77a792e8866e_0 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-260-1_kube-system_a6304778b09d1d4cc59e8feb77a74afa_1 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-260-1_kube-system_ebec19750ade6c9d4b1b9edd6e0d8e6e_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-260-1_kube-system_8268f6cdbfb1120c65b1a0df42e75741_0 2025-10-09 01:38:00,839 - DEBUG - [10.0.0.49]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-10-09 01:38:00,947 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.49/24 2025-10-09 01:38:00,948 - DEBUG - [10.0.0.49]: Running cmd : getent hosts 10.0.0.49 | head -n 1 | awk '{print $2}' 2025-10-09 01:38:01,074 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-260-1.cluster.local 2025-10-09 01:38:01,074 - DEBUG - [10.0.0.49]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-10-09 01:38:01,192 - DEBUG - Output : 10.0.0.49/24 2025-10-09 01:38:01,573 - INFO - Using existing project ['default-domain', 'k8s-default'](cbbe5478-dfd1-48ff-92f3-d257b43f1281) 2025-10-09 01:38:01,692 - INFO - Using existing project ['default-domain', 'k8s-default'](cbbe5478-dfd1-48ff-92f3-d257b43f1281) 2025-10-09 01:38:01,830 - INFO - ================================================================================ 2025-10-09 01:38:01,830 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-10-09 01:38:01,831 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-10-09 01:38:02,105 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 01:38:02,112 - DEBUG - Nothing to compare xmpp stats {'10.0.0.49': {'10.0.0.49': '1'}} with 2025-10-09 01:38:02,112 - INFO - Initial checks done. Running the testcase now 2025-10-09 01:38:02,112 - INFO - 2025-10-09 01:38:02,117 - INFO - Namespace default already exists 2025-10-09 01:38:02,117 - DEBUG - [10.0.0.49]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-9csct_kube-system_03af9417-4b03-4ab9-89dc-818d0acc99bd_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-10-09 01:38:02,299 - DEBUG - Output : cluster_project={} 2025-10-09 01:38:02,366 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-10-09 01:38:02,366 - DEBUG - Project uuid in Contrail is cbbe5478-dfd1-48ff-92f3-d257b43f1281 2025-10-09 01:38:02,366 - DEBUG - [10.0.0.49]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-9csct_kube-system_03af9417-4b03-4ab9-89dc-818d0acc99bd_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-10-09 01:38:02,534 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-10-09 01:38:02,535 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_NamespaceDatabaseList?namespace_uuid=df880cd8-cf51-4176-a622-ac4c1af77515 2025-10-09 01:38:02,539 - INFO - Namespace default with uuid df880cd8-cf51-4176-a622-ac4c1af77515 found in kube manager 2025-10-09 01:38:02,539 - INFO - Namespace default verification passed 2025-10-09 01:38:02,541 - DEBUG - Service ctest-nginx-svc-28917889 not present 2025-10-09 01:38:02,542 - INFO - Creating service ctest-nginx-svc-28917889 2025-10-09 01:38:02,552 - INFO - Created Service ctest-nginx-svc-28917889 2025-10-09 01:38:02,555 - DEBUG - Pod ctest-nginx-pod-17356574 not present 2025-10-09 01:38:02,556 - INFO - Creating Pod ctest-nginx-pod-17356574 2025-10-09 01:38:02,587 - DEBUG - Pod : ctest-nginx-pod-17356574 UUID is d3b7017d-8c12-4ea6-b10f-406a69a86f42 2025-10-09 01:38:02,590 - DEBUG - Pod ctest-nginx-pod-45027879 not present 2025-10-09 01:38:02,590 - INFO - Creating Pod ctest-nginx-pod-45027879 2025-10-09 01:38:02,618 - DEBUG - Pod : ctest-nginx-pod-45027879 UUID is 02a2f513-3eb7-448d-bb9b-786c33b9aa41 2025-10-09 01:38:03,023 - INFO - Created VN __public__, UUID :5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:03,294 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-09 01:38:03,299 - DEBUG - Requesting: http://10.0.0.49:8082/domain/a333fa80-09f9-4d02-917b-59de2eeff39e 2025-10-09 01:38:03,322 - DEBUG - Requesting: http://10.0.0.49:8082/project/cbbe5478-dfd1-48ff-92f3-d257b43f1281 2025-10-09 01:38:03,383 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:03,410 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:03,421 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/5e1830b7-b859-43c8-b462-d6420126f149 2025-10-09 01:38:03,435 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/5e1830b7-b859-43c8-b462-d6420126f149 2025-10-09 01:38:03,440 - DEBUG - Route Targets: [] 2025-10-09 01:38:03,440 - DEBUG - RT names not yet present for VN __public__ 2025-10-09 01:38:08,441 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-09 01:38:08,446 - DEBUG - Requesting: http://10.0.0.49:8082/domain/a333fa80-09f9-4d02-917b-59de2eeff39e 2025-10-09 01:38:08,452 - DEBUG - Requesting: http://10.0.0.49:8082/project/cbbe5478-dfd1-48ff-92f3-d257b43f1281 2025-10-09 01:38:08,506 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:08,515 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:08,522 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/5e1830b7-b859-43c8-b462-d6420126f149 2025-10-09 01:38:08,528 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/5e1830b7-b859-43c8-b462-d6420126f149 2025-10-09 01:38:08,532 - DEBUG - Requesting: http://10.0.0.49:8082/route-target/b603f19c-1986-43c4-a2ca-d9bad24310b0 2025-10-09 01:38:08,537 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-09 01:38:08,537 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:08,544 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/5e1830b7-b859-43c8-b462-d6420126f149 2025-10-09 01:38:08,554 - INFO - Verified VN network id 11 for VN 5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:08,554 - INFO - Verifications in API Server for VN __public__ passed 2025-10-09 01:38:08,554 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:08,561 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/5e1830b7-b859-43c8-b462-d6420126f149 2025-10-09 01:38:08,566 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/5e1830b7-b859-43c8-b462-d6420126f149 2025-10-09 01:38:08,570 - DEBUG - Requesting: http://10.0.0.49:8082/route-target/b603f19c-1986-43c4-a2ca-d9bad24310b0 2025-10-09 01:38:08,574 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-09 01:38:08,579 - DEBUG - Control-node 10.0.0.49 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6368367022814743119', 'uuid-lslong': '12325603633942860375'}, 'enable': 'true', 'created': '2025-10-09T01:38:02', 'last-modified': '2025-10-09T01:38:03', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.433492'} 2025-10-09 01:38:08,579 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-09 01:38:08,586 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-09 01:38:08,586 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-10-09 01:38:08,586 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-10-09 01:38:08,587 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-09 01:38:08,590 - DEBUG - Requesting: http://10.0.0.49:8082/domain/a333fa80-09f9-4d02-917b-59de2eeff39e 2025-10-09 01:38:08,595 - DEBUG - Requesting: http://10.0.0.49:8082/project/cbbe5478-dfd1-48ff-92f3-d257b43f1281 2025-10-09 01:38:08,626 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:08,639 - DEBUG - =>VN __public__ has no policy to be verified 2025-10-09 01:38:08,639 - DEBUG - Verifying the vn in opserver 2025-10-09 01:38:08,639 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.49 2025-10-09 01:38:08,640 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-networks 2025-10-09 01:38:08,644 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-10-09 01:38:08,644 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-10-09 01:38:08,644 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-09 01:38:08,649 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-09 01:38:08,653 - DEBUG - Do not have enough data to verify VN in agent 2025-10-09 01:38:08,653 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-09 01:38:08,657 - DEBUG - VRF ids for VN __public__: {} 2025-10-09 01:38:08,828 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-10-09 01:38:09,078 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-09 01:38:09,084 - DEBUG - Requesting: http://10.0.0.49:8082/domain/a333fa80-09f9-4d02-917b-59de2eeff39e 2025-10-09 01:38:09,091 - DEBUG - Requesting: http://10.0.0.49:8082/project/cbbe5478-dfd1-48ff-92f3-d257b43f1281 2025-10-09 01:38:09,131 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:09,144 - DEBUG - Requesting: http://10.0.0.49:8082/floating-ip-pool/1406dd12-5982-4bf7-b493-42c6acc23560 2025-10-09 01:38:09,150 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-09 01:38:09,154 - DEBUG - Requesting: http://10.0.0.49:8082/domain/a333fa80-09f9-4d02-917b-59de2eeff39e 2025-10-09 01:38:09,159 - DEBUG - Requesting: http://10.0.0.49:8082/project/cbbe5478-dfd1-48ff-92f3-d257b43f1281 2025-10-09 01:38:09,186 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/5860fc10-f22f-4a4f-ab0d-5733836d9257 2025-10-09 01:38:09,202 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-10-09 01:38:09,202 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-10-09 01:38:09,208 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-09 01:38:09,213 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-10-09 01:38:09,213 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-10-09 01:38:09,216 - DEBUG - Ingress ctest-nginx-ingress-85615496 not present 2025-10-09 01:38:09,217 - INFO - Creating Ingress ctest-nginx-ingress-85615496 2025-10-09 01:38:09,224 - INFO - Created Ingress ctest-nginx-ingress-85615496 2025-10-09 01:38:09,227 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-85615496 2025-10-09 01:38:12,233 - DEBUG - For Ingress ctest-nginx-ingress-85615496, Cluster IP: 10.47.255.248, External IPs ['162.226.118.252'] 2025-10-09 01:38:12,233 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-85615496 2025-10-09 01:38:12,233 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=b4077e7e-da91-45ef-95fd-2f51455ceacc 2025-10-09 01:38:12,239 - INFO - Ingress ctest-nginx-ingress-85615496 with uuid b4077e7e-da91-45ef-95fd-2f51455ceacc found in kube manager 2025-10-09 01:38:12,240 - INFO - Ingress ctest-nginx-ingress-85615496 verification passed 2025-10-09 01:38:12,243 - DEBUG - Pod ctest-busybox-pod-15715044 not present 2025-10-09 01:38:12,243 - INFO - Creating Pod ctest-busybox-pod-15715044 2025-10-09 01:38:12,274 - DEBUG - Pod : ctest-busybox-pod-15715044 UUID is 6fd65e39-a459-49df-8084-4ba157e2fa70 2025-10-09 01:38:12,284 - DEBUG - Pod ctest-nginx-pod-17356574 not in running state.Currently in Pending 2025-10-09 01:38:17,292 - DEBUG - Pod ctest-nginx-pod-17356574 not in running state.Currently in Pending 2025-10-09 01:38:22,302 - INFO - Pod ctest-nginx-pod-17356574 is in running state.Got IP 10.47.255.250 2025-10-09 01:38:22,374 - DEBUG - Pod ctest-nginx-pod-17356574 has vmi 99e4e054-a4b0-11f0-8a86-02428c2328f8 2025-10-09 01:38:22,374 - INFO - Verified pod ctest-nginx-pod-17356574 in contrail-api 2025-10-09 01:38:22,374 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-09 01:38:22,386 - DEBUG - VMI 99e4e054-a4b0-11f0-8a86-02428c2328f8 is active in agent 10.0.0.49 2025-10-09 01:38:22,386 - INFO - Verified Pod ctest-nginx-pod-17356574 in agent 10.0.0.49 2025-10-09 01:38:22,386 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=d3b7017d-8c12-4ea6-b10f-406a69a86f42 2025-10-09 01:38:22,390 - INFO - Pod ctest-nginx-pod-17356574 with uuid d3b7017d-8c12-4ea6-b10f-406a69a86f42 found in kube manager 2025-10-09 01:38:22,390 - INFO - Pod ctest-nginx-pod-17356574 verification passed 2025-10-09 01:38:22,483 - DEBUG - [Pod ctest-nginx-pod-17356574] Cmd: echo ctest-nginx-pod-17356574 > /usr/share/nginx/html/index.html, Output: 2025-10-09 01:38:22,491 - INFO - Pod ctest-nginx-pod-45027879 is in running state.Got IP 10.47.255.249 2025-10-09 01:38:22,547 - DEBUG - Pod ctest-nginx-pod-45027879 has vmi 9a94db12-a4b0-11f0-8a86-02428c2328f8 2025-10-09 01:38:22,547 - INFO - Verified pod ctest-nginx-pod-45027879 in contrail-api 2025-10-09 01:38:22,547 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-09 01:38:22,558 - DEBUG - VMI 9a94db12-a4b0-11f0-8a86-02428c2328f8 is active in agent 10.0.0.49 2025-10-09 01:38:22,559 - INFO - Verified Pod ctest-nginx-pod-45027879 in agent 10.0.0.49 2025-10-09 01:38:22,559 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=02a2f513-3eb7-448d-bb9b-786c33b9aa41 2025-10-09 01:38:22,564 - INFO - Pod ctest-nginx-pod-45027879 with uuid 02a2f513-3eb7-448d-bb9b-786c33b9aa41 found in kube manager 2025-10-09 01:38:22,564 - INFO - Pod ctest-nginx-pod-45027879 verification passed 2025-10-09 01:38:22,644 - DEBUG - [Pod ctest-nginx-pod-45027879] Cmd: echo ctest-nginx-pod-45027879 > /usr/share/nginx/html/index.html, Output: 2025-10-09 01:38:22,653 - DEBUG - Pod ctest-busybox-pod-15715044 not in running state.Currently in Pending 2025-10-09 01:38:27,660 - INFO - Pod ctest-busybox-pod-15715044 is in running state.Got IP 10.47.255.247 2025-10-09 01:38:27,713 - DEBUG - Pod ctest-busybox-pod-15715044 has vmi 9f52416c-a4b0-11f0-8a86-02428c2328f8 2025-10-09 01:38:27,713 - INFO - Verified pod ctest-busybox-pod-15715044 in contrail-api 2025-10-09 01:38:27,713 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-09 01:38:27,721 - DEBUG - VMI 9f52416c-a4b0-11f0-8a86-02428c2328f8 is active in agent 10.0.0.49 2025-10-09 01:38:27,721 - INFO - Verified Pod ctest-busybox-pod-15715044 in agent 10.0.0.49 2025-10-09 01:38:27,721 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=6fd65e39-a459-49df-8084-4ba157e2fa70 2025-10-09 01:38:27,725 - INFO - Pod ctest-busybox-pod-15715044 with uuid 6fd65e39-a459-49df-8084-4ba157e2fa70 found in kube manager 2025-10-09 01:38:27,725 - INFO - Pod ctest-busybox-pod-15715044 verification passed 2025-10-09 01:38:27,797 - DEBUG - [Pod ctest-busybox-pod-15715044] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-45027879 2025-10-09 01:38:27,797 - DEBUG - [Pod Pod ctest-busybox-pod-15715044] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-10-09 01:38:27,797 - DEBUG - [Pod Pod ctest-busybox-pod-15715044] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-45027879 2025-10-09 01:38:27,871 - DEBUG - [Pod ctest-busybox-pod-15715044] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-17356574 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-09 01:38:27,871 - DEBUG - [Pod Pod ctest-busybox-pod-15715044] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-10-09 01:38:27,871 - DEBUG - [Pod Pod ctest-busybox-pod-15715044] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-17356574 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-09 01:38:27,871 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-17356574': 1, 'ctest-nginx-pod-45027879': 1} 2025-10-09 01:38:27,871 - WARNING - Nothing to delete parallely 2025-10-09 01:38:27,871 - INFO - Deleting pod default:ctest-busybox-pod-15715044 2025-10-09 01:38:27,886 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=6fd65e39-a459-49df-8084-4ba157e2fa70 2025-10-09 01:38:27,897 - WARNING - Pod uuid 6fd65e39-a459-49df-8084-4ba157e2fa70 is still seen in agent 10.0.0.49 VM list 2025-10-09 01:38:32,898 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=6fd65e39-a459-49df-8084-4ba157e2fa70 2025-10-09 01:38:32,904 - WARNING - Pod uuid 6fd65e39-a459-49df-8084-4ba157e2fa70 is still seen in agent 10.0.0.49 VM list 2025-10-09 01:38:37,905 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=6fd65e39-a459-49df-8084-4ba157e2fa70 2025-10-09 01:38:37,911 - WARNING - Pod uuid 6fd65e39-a459-49df-8084-4ba157e2fa70 is still seen in agent 10.0.0.49 VM list 2025-10-09 01:38:42,912 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=6fd65e39-a459-49df-8084-4ba157e2fa70 2025-10-09 01:38:42,917 - WARNING - Pod uuid 6fd65e39-a459-49df-8084-4ba157e2fa70 is still seen in agent 10.0.0.49 VM list 2025-10-09 01:38:47,918 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=6fd65e39-a459-49df-8084-4ba157e2fa70 2025-10-09 01:38:47,926 - WARNING - Pod uuid 6fd65e39-a459-49df-8084-4ba157e2fa70 is still seen in agent 10.0.0.49 VM list 2025-10-09 01:38:52,927 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=6fd65e39-a459-49df-8084-4ba157e2fa70 2025-10-09 01:38:52,932 - WARNING - Pod uuid 6fd65e39-a459-49df-8084-4ba157e2fa70 is still seen in agent 10.0.0.49 VM list 2025-10-09 01:38:57,933 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=6fd65e39-a459-49df-8084-4ba157e2fa70 2025-10-09 01:38:57,939 - WARNING - Pod uuid 6fd65e39-a459-49df-8084-4ba157e2fa70 is still seen in agent 10.0.0.49 VM list 2025-10-09 01:39:02,940 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=6fd65e39-a459-49df-8084-4ba157e2fa70 2025-10-09 01:39:02,946 - DEBUG - Pod 6fd65e39-a459-49df-8084-4ba157e2fa70 is not in agent 10.0.0.49 VM list 2025-10-09 01:39:02,946 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9f52416c-a4b0-11f0-8a86-02428c2328f8 2025-10-09 01:39:02,952 - DEBUG - VMI 9f52416c-a4b0-11f0-8a86-02428c2328f8 is removed from agent 10.0.0.49 2025-10-09 01:39:02,953 - INFO - Verified that pod ctest-busybox-pod-15715044 is removed in agent 2025-10-09 01:39:02,953 - INFO - Deleting Ingress : ctest-nginx-ingress-85615496 2025-10-09 01:39:02,960 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=b4077e7e-da91-45ef-95fd-2f51455ceacc 2025-10-09 01:39:02,963 - ERROR - Ingress ctest-nginx-ingress-85615496 with uuid b4077e7e-da91-45ef-95fd-2f51455ceacc still found in kube manager 2025-10-09 01:39:03,964 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=b4077e7e-da91-45ef-95fd-2f51455ceacc 2025-10-09 01:39:03,971 - INFO - Ingress ctest-nginx-ingress-85615496 with uuid b4077e7e-da91-45ef-95fd-2f51455ceacc deleted successfully from kube manager 2025-10-09 01:39:03,971 - INFO - Verifications on Ingress %s cleanup passed 2025-10-09 01:39:03,971 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-10-09 01:39:04,137 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-09 01:39:04,147 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-10-09 01:39:04,147 - INFO - Deleting VN __public__ 2025-10-09 01:39:04,515 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/5e1830b7-b859-43c8-b462-d6420126f149 2025-10-09 01:39:04,530 - DEBUG - Response Code: 404 2025-10-09 01:39:04,530 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-10-09 01:39:04,541 - DEBUG - Requesting: http://10.0.0.49:8082/domain/a333fa80-09f9-4d02-917b-59de2eeff39e 2025-10-09 01:39:04,555 - DEBUG - Requesting: http://10.0.0.49:8082/project/cbbe5478-dfd1-48ff-92f3-d257b43f1281 2025-10-09 01:39:04,622 - INFO - Validated that VN __public__ is not found in API Server 2025-10-09 01:39:04,622 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-10-09 01:39:04,628 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-09 01:39:04,635 - DEBUG - VN __public__ is not present in Agent 10.0.0.49 2025-10-09 01:39:04,635 - INFO - Validated that VN __public__ is not in any agent 2025-10-09 01:39:04,635 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-09 01:39:04,643 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-09 01:39:04,666 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-09 01:39:04,670 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-10-09 01:39:04,670 - INFO - Deleting pod default:ctest-nginx-pod-45027879 2025-10-09 01:39:04,682 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=02a2f513-3eb7-448d-bb9b-786c33b9aa41 2025-10-09 01:39:04,688 - WARNING - Pod uuid 02a2f513-3eb7-448d-bb9b-786c33b9aa41 is still seen in agent 10.0.0.49 VM list 2025-10-09 01:39:09,689 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=02a2f513-3eb7-448d-bb9b-786c33b9aa41 2025-10-09 01:39:09,693 - DEBUG - Pod 02a2f513-3eb7-448d-bb9b-786c33b9aa41 is not in agent 10.0.0.49 VM list 2025-10-09 01:39:09,693 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9a94db12-a4b0-11f0-8a86-02428c2328f8 2025-10-09 01:39:09,698 - DEBUG - VMI 9a94db12-a4b0-11f0-8a86-02428c2328f8 is removed from agent 10.0.0.49 2025-10-09 01:39:09,698 - INFO - Verified that pod ctest-nginx-pod-45027879 is removed in agent 2025-10-09 01:39:09,698 - INFO - Deleting pod default:ctest-nginx-pod-17356574 2025-10-09 01:39:09,715 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=d3b7017d-8c12-4ea6-b10f-406a69a86f42 2025-10-09 01:39:09,722 - WARNING - Pod uuid d3b7017d-8c12-4ea6-b10f-406a69a86f42 is still seen in agent 10.0.0.49 VM list 2025-10-09 01:39:14,722 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=d3b7017d-8c12-4ea6-b10f-406a69a86f42 2025-10-09 01:39:14,728 - DEBUG - Pod d3b7017d-8c12-4ea6-b10f-406a69a86f42 is not in agent 10.0.0.49 VM list 2025-10-09 01:39:14,728 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:99e4e054-a4b0-11f0-8a86-02428c2328f8 2025-10-09 01:39:14,732 - DEBUG - VMI 99e4e054-a4b0-11f0-8a86-02428c2328f8 is removed from agent 10.0.0.49 2025-10-09 01:39:14,732 - INFO - Verified that pod ctest-nginx-pod-17356574 is removed in agent 2025-10-09 01:39:14,732 - INFO - Deleting service : ctest-nginx-svc-28917889 2025-10-09 01:39:15,273 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 01:39:15,278 - DEBUG - No XMPP flaps were noticed during the test