2025-12-22 01:39:45,595 - DEBUG - [10.0.0.249]: Running cmd : hostname 2025-12-22 01:39:45,867 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-281-1 2025-12-22 01:39:45,867 - DEBUG - [10.0.0.249]: Running cmd : hostname -f 2025-12-22 01:39:46,000 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-281-1.cluster.local 2025-12-22 01:39:46,000 - DEBUG - [10.0.0.249]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-12-22 01:39:46,176 - DEBUG - Output : NAMES contrail_test_CrJ72d3pj k8s_autoscaler_dns-autoscaler-595558c478-wtbcf_kube-system_97aa0e88-ae24-46b8-8940-90bc29b6c94a_0 k8s_coredns_coredns-77f7cc69db-tmwgn_kube-system_1fe839bb-c903-4138-9417-5ec3a16108be_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-27rjj_kube-system_f7ab9535-fa53-4dfe-b245-976e34f352b4_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-27rjj_kube-system_f7ab9535-fa53-4dfe-b245-976e34f352b4_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-gcfch_kube-system_8d863cef-3c99-4d9f-b6b2-fe27b17321ee_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-27rjj_kube-system_f7ab9535-fa53-4dfe-b245-976e34f352b4_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-8fwk2_kube-system_1d423f86-78cc-4d56-abde-2491b92cc0f6_0 k8s_opensdn-controller-control-named_opensdn-controller-control-8fwk2_kube-system_1d423f86-78cc-4d56-abde-2491b92cc0f6_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-nsnll_kube-system_2cfade64-f7f8-4e18-b2d8-e88799fce213_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-nsnll_kube-system_2cfade64-f7f8-4e18-b2d8-e88799fce213_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-flfgp_kube-system_e232646d-4446-4245-ac5d-a0f3adbc5030_0 k8s_opensdn-analytics-collector_opensdn-analytics-flfgp_kube-system_e232646d-4446-4245-ac5d-a0f3adbc5030_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-bcqn2_kube-system_ae5092d1-d9e8-494d-9f8c-65cf3706b23c_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-bcqn2_kube-system_ae5092d1-d9e8-494d-9f8c-65cf3706b23c_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-bprzv_kube-system_3b48b77e-fe04-48d1-a8bd-3bdb583d85ce_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-gcfch_kube-system_8d863cef-3c99-4d9f-b6b2-fe27b17321ee_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-27rjj_kube-system_f7ab9535-fa53-4dfe-b245-976e34f352b4_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-8fwk2_kube-system_1d423f86-78cc-4d56-abde-2491b92cc0f6_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-kbxjj_kube-system_4487a8df-ed7a-4cba-baee-9a860971713b_0 k8s_opensdn-agent-nodemgr_opensdn-agent-9d9rc_kube-system_2c10b2d7-67b2-4426-b038-e44ffdd3e6fa_0 k8s_kafka_opensdn-analytics-alarm-nsnll_kube-system_2cfade64-f7f8-4e18-b2d8-e88799fce213_0 k8s_opensdn-analytics-api_opensdn-analytics-flfgp_kube-system_e232646d-4446-4245-ac5d-a0f3adbc5030_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-bcqn2_kube-system_ae5092d1-d9e8-494d-9f8c-65cf3706b23c_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-bprzv_kube-system_3b48b77e-fe04-48d1-a8bd-3bdb583d85ce_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-gcfch_kube-system_8d863cef-3c99-4d9f-b6b2-fe27b17321ee_0 k8s_opensdn-controller-config-api_opensdn-controller-config-27rjj_kube-system_f7ab9535-fa53-4dfe-b245-976e34f352b4_0 k8s_opensdn-controller-control_opensdn-controller-control-8fwk2_kube-system_1d423f86-78cc-4d56-abde-2491b92cc0f6_0 k8s_rabbitmq_rabbitmq-nl8w6_kube-system_22ea8d96-f79c-4062-8b34-8a7a129a7bec_0 k8s_redis_redis-hzdps_kube-system_7edbd267-00f0-42a3-ad39-11163ea57ac8_0 k8s_config-zookeeper_config-zookeeper-5dgxc_kube-system_501c0902-3bc3-4c23-941b-5e910121a9b9_0 k8s_opensdn-configdb_opensdn-configdb-kbxjj_kube-system_4487a8df-ed7a-4cba-baee-9a860971713b_0 k8s_opensdn-vrouter-agent_opensdn-agent-9d9rc_kube-system_2c10b2d7-67b2-4426-b038-e44ffdd3e6fa_0 k8s_opensdn-kube-manager_opensdn-kube-manager-8zvdv_kube-system_4797f29c-2ee6-4c63-97fe-72b07241bf4c_0 k8s_kube-proxy_kube-proxy-ghj56_kube-system_29028552-aff8-4a9c-b737-3e146e2fe89a_0 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-281-1_kube-system_bdd4b9d4a639c98e3b2acc5e065fdfd2_1 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-281-1_kube-system_05341e04ba05144ba752d48aa9e4286d_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-281-1_kube-system_46e21984840e666a668bf1d3c971abfa_0 2025-12-22 01:39:46,176 - DEBUG - [10.0.0.249]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-12-22 01:39:46,319 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.3/32 10.96.0.1/32 10.0.0.249/24 2025-12-22 01:39:46,319 - DEBUG - [10.0.0.249]: Running cmd : getent hosts 10.0.0.249 | head -n 1 | awk '{print $2}' 2025-12-22 01:39:46,450 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-281-1.cluster.local 2025-12-22 01:39:46,450 - DEBUG - [10.0.0.249]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-12-22 01:39:46,587 - DEBUG - Output : 10.0.0.249/24 2025-12-22 01:39:46,965 - INFO - Using existing project ['default-domain', 'k8s-default'](6a507901-d7f1-454b-aeff-0cb7f1afa9ed) 2025-12-22 01:39:47,087 - INFO - Using existing project ['default-domain', 'k8s-default'](6a507901-d7f1-454b-aeff-0cb7f1afa9ed) 2025-12-22 01:39:47,230 - INFO - ================================================================================ 2025-12-22 01:39:47,230 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-12-22 01:39:47,230 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-12-22 01:39:47,525 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq? 2025-12-22 01:39:47,531 - DEBUG - Nothing to compare xmpp stats {'10.0.0.249': {'10.0.0.249': '2'}} with 2025-12-22 01:39:47,531 - INFO - Initial checks done. Running the testcase now 2025-12-22 01:39:47,531 - INFO - 2025-12-22 01:39:47,536 - INFO - Namespace default already exists 2025-12-22 01:39:47,537 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-8zvdv_kube-system_4797f29c-2ee6-4c63-97fe-72b07241bf4c_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-12-22 01:39:47,720 - DEBUG - Output : cluster_project={} 2025-12-22 01:39:47,803 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-12-22 01:39:47,804 - DEBUG - Project uuid in Contrail is 6a507901-d7f1-454b-aeff-0cb7f1afa9ed 2025-12-22 01:39:47,804 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-8zvdv_kube-system_4797f29c-2ee6-4c63-97fe-72b07241bf4c_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-12-22 01:39:47,973 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-12-22 01:39:47,973 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_NamespaceDatabaseList?namespace_uuid=8b96593d-a029-496c-8d35-f19f1e917b96 2025-12-22 01:39:47,977 - INFO - Namespace default with uuid 8b96593d-a029-496c-8d35-f19f1e917b96 found in kube manager 2025-12-22 01:39:47,977 - INFO - Namespace default verification passed 2025-12-22 01:39:47,981 - DEBUG - Service ctest-nginx-svc-33420566 not present 2025-12-22 01:39:47,982 - INFO - Creating service ctest-nginx-svc-33420566 2025-12-22 01:39:47,994 - INFO - Created Service ctest-nginx-svc-33420566 2025-12-22 01:39:48,002 - DEBUG - Pod ctest-nginx-pod-70124969 not present 2025-12-22 01:39:48,003 - INFO - Creating Pod ctest-nginx-pod-70124969 2025-12-22 01:39:48,056 - DEBUG - Pod : ctest-nginx-pod-70124969 UUID is 256e6dad-3264-4057-a876-574cd03ede02 2025-12-22 01:39:48,061 - DEBUG - Pod ctest-nginx-pod-63079020 not present 2025-12-22 01:39:48,061 - INFO - Creating Pod ctest-nginx-pod-63079020 2025-12-22 01:39:48,088 - DEBUG - Pod : ctest-nginx-pod-63079020 UUID is 1c1752cb-4ee6-41fb-8cbc-e4a6843cc2bf 2025-12-22 01:39:48,491 - INFO - Created VN __public__, UUID :07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:48,808 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-22 01:39:48,815 - DEBUG - Requesting: http://10.0.0.249:8082/domain/382747db-e7b4-4bbc-b2c8-0d9c92957c0e 2025-12-22 01:39:48,835 - DEBUG - Requesting: http://10.0.0.249:8082/project/6a507901-d7f1-454b-aeff-0cb7f1afa9ed 2025-12-22 01:39:48,886 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:48,895 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:48,904 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/5b3ed7a9-831b-448f-9bc0-7c7498c2e02b 2025-12-22 01:39:48,918 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/5b3ed7a9-831b-448f-9bc0-7c7498c2e02b 2025-12-22 01:39:48,926 - DEBUG - Route Targets: [] 2025-12-22 01:39:48,926 - DEBUG - RT names not yet present for VN __public__ 2025-12-22 01:39:53,926 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-22 01:39:53,933 - DEBUG - Requesting: http://10.0.0.249:8082/domain/382747db-e7b4-4bbc-b2c8-0d9c92957c0e 2025-12-22 01:39:53,947 - DEBUG - Requesting: http://10.0.0.249:8082/project/6a507901-d7f1-454b-aeff-0cb7f1afa9ed 2025-12-22 01:39:53,982 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:53,993 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:54,002 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/5b3ed7a9-831b-448f-9bc0-7c7498c2e02b 2025-12-22 01:39:54,011 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/5b3ed7a9-831b-448f-9bc0-7c7498c2e02b 2025-12-22 01:39:54,017 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/72288246-04eb-4c1d-8ebb-ba8989e380ae 2025-12-22 01:39:54,024 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-22 01:39:54,024 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:54,032 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/5b3ed7a9-831b-448f-9bc0-7c7498c2e02b 2025-12-22 01:39:54,047 - INFO - Verified VN network id 11 for VN 07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:54,047 - INFO - Verifications in API Server for VN __public__ passed 2025-12-22 01:39:54,047 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:54,056 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/5b3ed7a9-831b-448f-9bc0-7c7498c2e02b 2025-12-22 01:39:54,062 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/5b3ed7a9-831b-448f-9bc0-7c7498c2e02b 2025-12-22 01:39:54,068 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/72288246-04eb-4c1d-8ebb-ba8989e380ae 2025-12-22 01:39:54,073 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-12-22 01:39:54,078 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '560543747055829049', 'uuid-lslong': '13280198907507789871'}, 'enable': 'true', 'created': '2025-12-22T01:39:48', 'last-modified': '2025-12-22T01:39:48', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.455780'} 2025-12-22 01:39:54,079 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-22 01:39:54,087 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-22 01:39:54,087 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-12-22 01:39:54,087 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-12-22 01:39:54,087 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-22 01:39:54,093 - DEBUG - Requesting: http://10.0.0.249:8082/domain/382747db-e7b4-4bbc-b2c8-0d9c92957c0e 2025-12-22 01:39:54,100 - DEBUG - Requesting: http://10.0.0.249:8082/project/6a507901-d7f1-454b-aeff-0cb7f1afa9ed 2025-12-22 01:39:54,134 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:54,145 - DEBUG - =>VN __public__ has no policy to be verified 2025-12-22 01:39:54,145 - DEBUG - Verifying the vn in opserver 2025-12-22 01:39:54,145 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.249 2025-12-22 01:39:54,146 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2025-12-22 01:39:54,151 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-12-22 01:39:54,151 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-12-22 01:39:54,151 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-22 01:39:54,158 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-22 01:39:54,163 - DEBUG - Do not have enough data to verify VN in agent 2025-12-22 01:39:54,163 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-22 01:39:54,167 - DEBUG - VRF ids for VN __public__: {} 2025-12-22 01:39:54,336 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-12-22 01:39:54,540 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-22 01:39:54,547 - DEBUG - Requesting: http://10.0.0.249:8082/domain/382747db-e7b4-4bbc-b2c8-0d9c92957c0e 2025-12-22 01:39:54,555 - DEBUG - Requesting: http://10.0.0.249:8082/project/6a507901-d7f1-454b-aeff-0cb7f1afa9ed 2025-12-22 01:39:54,607 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:54,629 - DEBUG - Requesting: http://10.0.0.249:8082/floating-ip-pool/b1235337-41e7-4b55-90f8-925e5e02bfe0 2025-12-22 01:39:54,638 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-22 01:39:54,645 - DEBUG - Requesting: http://10.0.0.249:8082/domain/382747db-e7b4-4bbc-b2c8-0d9c92957c0e 2025-12-22 01:39:54,654 - DEBUG - Requesting: http://10.0.0.249:8082/project/6a507901-d7f1-454b-aeff-0cb7f1afa9ed 2025-12-22 01:39:54,687 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07c77391-6c54-4039-b84c-be8b7c88502f 2025-12-22 01:39:54,709 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-12-22 01:39:54,709 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-12-22 01:39:54,717 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-12-22 01:39:54,725 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-12-22 01:39:54,725 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-12-22 01:39:54,730 - DEBUG - Ingress ctest-nginx-ingress-10459662 not present 2025-12-22 01:39:54,731 - INFO - Creating Ingress ctest-nginx-ingress-10459662 2025-12-22 01:39:54,744 - INFO - Created Ingress ctest-nginx-ingress-10459662 2025-12-22 01:39:54,752 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-10459662 2025-12-22 01:39:57,760 - DEBUG - For Ingress ctest-nginx-ingress-10459662, Cluster IP: 10.47.255.248, External IPs ['33.104.187.60'] 2025-12-22 01:39:57,760 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-10459662 2025-12-22 01:39:57,761 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=d5e721e1-34be-4769-841e-3786c008e060 2025-12-22 01:39:57,770 - INFO - Ingress ctest-nginx-ingress-10459662 with uuid d5e721e1-34be-4769-841e-3786c008e060 found in kube manager 2025-12-22 01:39:57,770 - INFO - Ingress ctest-nginx-ingress-10459662 verification passed 2025-12-22 01:39:57,776 - DEBUG - Pod ctest-busybox-pod-10560891 not present 2025-12-22 01:39:57,777 - INFO - Creating Pod ctest-busybox-pod-10560891 2025-12-22 01:39:57,812 - DEBUG - Pod : ctest-busybox-pod-10560891 UUID is fc90a930-9d17-406b-a8be-0a18bd13a5f2 2025-12-22 01:39:57,824 - DEBUG - Pod ctest-nginx-pod-70124969 not in running state.Currently in Pending 2025-12-22 01:40:02,832 - INFO - Pod ctest-nginx-pod-70124969 is in running state.Got IP 10.47.255.250 2025-12-22 01:40:02,924 - DEBUG - Pod ctest-nginx-pod-70124969 has vmi 1b7377a6-ded7-11f0-b5dc-024248d2bb15 2025-12-22 01:40:02,925 - INFO - Verified pod ctest-nginx-pod-70124969 in contrail-api 2025-12-22 01:40:02,925 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-12-22 01:40:02,944 - DEBUG - VMI 1b7377a6-ded7-11f0-b5dc-024248d2bb15 is active in agent 10.0.0.249 2025-12-22 01:40:02,944 - INFO - Verified Pod ctest-nginx-pod-70124969 in agent 10.0.0.249 2025-12-22 01:40:02,944 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=256e6dad-3264-4057-a876-574cd03ede02 2025-12-22 01:40:02,948 - INFO - Pod ctest-nginx-pod-70124969 with uuid 256e6dad-3264-4057-a876-574cd03ede02 found in kube manager 2025-12-22 01:40:02,948 - INFO - Pod ctest-nginx-pod-70124969 verification passed 2025-12-22 01:40:03,076 - DEBUG - [Pod ctest-nginx-pod-70124969] Cmd: echo ctest-nginx-pod-70124969 > /usr/share/nginx/html/index.html, Output: 2025-12-22 01:40:03,087 - INFO - Pod ctest-nginx-pod-63079020 is in running state.Got IP 10.47.255.249 2025-12-22 01:40:03,147 - DEBUG - Pod ctest-nginx-pod-63079020 has vmi 1c25db80-ded7-11f0-b5dc-024248d2bb15 2025-12-22 01:40:03,147 - INFO - Verified pod ctest-nginx-pod-63079020 in contrail-api 2025-12-22 01:40:03,147 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-12-22 01:40:03,171 - DEBUG - VMI 1c25db80-ded7-11f0-b5dc-024248d2bb15 is active in agent 10.0.0.249 2025-12-22 01:40:03,172 - INFO - Verified Pod ctest-nginx-pod-63079020 in agent 10.0.0.249 2025-12-22 01:40:03,172 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=1c1752cb-4ee6-41fb-8cbc-e4a6843cc2bf 2025-12-22 01:40:03,180 - INFO - Pod ctest-nginx-pod-63079020 with uuid 1c1752cb-4ee6-41fb-8cbc-e4a6843cc2bf found in kube manager 2025-12-22 01:40:03,180 - INFO - Pod ctest-nginx-pod-63079020 verification passed 2025-12-22 01:40:03,390 - DEBUG - [Pod ctest-nginx-pod-63079020] Cmd: echo ctest-nginx-pod-63079020 > /usr/share/nginx/html/index.html, Output: 2025-12-22 01:40:03,400 - DEBUG - Pod ctest-busybox-pod-10560891 not in running state.Currently in Pending 2025-12-22 01:40:08,408 - DEBUG - Pod ctest-busybox-pod-10560891 not in running state.Currently in Pending 2025-12-22 01:40:13,417 - INFO - Pod ctest-busybox-pod-10560891 is in running state.Got IP 10.47.255.247 2025-12-22 01:40:13,470 - DEBUG - Pod ctest-busybox-pod-10560891 has vmi 21816388-ded7-11f0-b5dc-024248d2bb15 2025-12-22 01:40:13,471 - INFO - Verified pod ctest-busybox-pod-10560891 in contrail-api 2025-12-22 01:40:13,471 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-12-22 01:40:13,481 - DEBUG - VMI 21816388-ded7-11f0-b5dc-024248d2bb15 is active in agent 10.0.0.249 2025-12-22 01:40:13,482 - INFO - Verified Pod ctest-busybox-pod-10560891 in agent 10.0.0.249 2025-12-22 01:40:13,482 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=fc90a930-9d17-406b-a8be-0a18bd13a5f2 2025-12-22 01:40:13,486 - INFO - Pod ctest-busybox-pod-10560891 with uuid fc90a930-9d17-406b-a8be-0a18bd13a5f2 found in kube manager 2025-12-22 01:40:13,486 - INFO - Pod ctest-busybox-pod-10560891 verification passed 2025-12-22 01:40:13,588 - DEBUG - [Pod ctest-busybox-pod-10560891] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-70124969 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-12-22 01:40:13,589 - DEBUG - [Pod Pod ctest-busybox-pod-10560891] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-12-22 01:40:13,589 - DEBUG - [Pod Pod ctest-busybox-pod-10560891] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-70124969 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-12-22 01:40:13,678 - DEBUG - [Pod ctest-busybox-pod-10560891] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-70124969 - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-12-22 01:40:13,678 - DEBUG - [Pod Pod ctest-busybox-pod-10560891] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-12-22 01:40:13,678 - DEBUG - [Pod Pod ctest-busybox-pod-10560891] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-70124969 - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-12-22 01:40:13,762 - DEBUG - [Pod ctest-busybox-pod-10560891] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% ctest-nginx-pod-63079020 |********************************| 25 0:00:00 ETA written to stdout 2025-12-22 01:40:13,762 - DEBUG - [Pod Pod ctest-busybox-pod-10560891] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-12-22 01:40:13,762 - DEBUG - [Pod Pod ctest-busybox-pod-10560891] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% ctest-nginx-pod-63079020 |********************************| 25 0:00:00 ETA written to stdout 2025-12-22 01:40:13,762 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-70124969': 2, 'ctest-nginx-pod-63079020': 1} 2025-12-22 01:40:13,763 - WARNING - Nothing to delete parallely 2025-12-22 01:40:13,763 - INFO - Deleting pod default:ctest-busybox-pod-10560891 2025-12-22 01:40:13,778 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=fc90a930-9d17-406b-a8be-0a18bd13a5f2 2025-12-22 01:40:13,784 - WARNING - Pod uuid fc90a930-9d17-406b-a8be-0a18bd13a5f2 is still seen in agent 10.0.0.249 VM list 2025-12-22 01:40:18,785 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=fc90a930-9d17-406b-a8be-0a18bd13a5f2 2025-12-22 01:40:18,792 - WARNING - Pod uuid fc90a930-9d17-406b-a8be-0a18bd13a5f2 is still seen in agent 10.0.0.249 VM list 2025-12-22 01:40:23,793 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=fc90a930-9d17-406b-a8be-0a18bd13a5f2 2025-12-22 01:40:23,800 - WARNING - Pod uuid fc90a930-9d17-406b-a8be-0a18bd13a5f2 is still seen in agent 10.0.0.249 VM list 2025-12-22 01:40:28,801 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=fc90a930-9d17-406b-a8be-0a18bd13a5f2 2025-12-22 01:40:28,808 - WARNING - Pod uuid fc90a930-9d17-406b-a8be-0a18bd13a5f2 is still seen in agent 10.0.0.249 VM list 2025-12-22 01:40:33,809 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=fc90a930-9d17-406b-a8be-0a18bd13a5f2 2025-12-22 01:40:33,815 - WARNING - Pod uuid fc90a930-9d17-406b-a8be-0a18bd13a5f2 is still seen in agent 10.0.0.249 VM list 2025-12-22 01:40:38,816 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=fc90a930-9d17-406b-a8be-0a18bd13a5f2 2025-12-22 01:40:38,821 - WARNING - Pod uuid fc90a930-9d17-406b-a8be-0a18bd13a5f2 is still seen in agent 10.0.0.249 VM list 2025-12-22 01:40:43,822 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=fc90a930-9d17-406b-a8be-0a18bd13a5f2 2025-12-22 01:40:43,828 - WARNING - Pod uuid fc90a930-9d17-406b-a8be-0a18bd13a5f2 is still seen in agent 10.0.0.249 VM list 2025-12-22 01:40:48,829 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=fc90a930-9d17-406b-a8be-0a18bd13a5f2 2025-12-22 01:40:48,835 - DEBUG - Pod fc90a930-9d17-406b-a8be-0a18bd13a5f2 is not in agent 10.0.0.249 VM list 2025-12-22 01:40:48,835 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:21816388-ded7-11f0-b5dc-024248d2bb15 2025-12-22 01:40:48,839 - DEBUG - VMI 21816388-ded7-11f0-b5dc-024248d2bb15 is removed from agent 10.0.0.249 2025-12-22 01:40:48,840 - INFO - Verified that pod ctest-busybox-pod-10560891 is removed in agent 2025-12-22 01:40:48,840 - INFO - Deleting Ingress : ctest-nginx-ingress-10459662 2025-12-22 01:40:48,857 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=d5e721e1-34be-4769-841e-3786c008e060 2025-12-22 01:40:48,861 - ERROR - Ingress ctest-nginx-ingress-10459662 with uuid d5e721e1-34be-4769-841e-3786c008e060 still found in kube manager 2025-12-22 01:40:49,862 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=d5e721e1-34be-4769-841e-3786c008e060 2025-12-22 01:40:49,876 - ERROR - Ingress ctest-nginx-ingress-10459662 with uuid d5e721e1-34be-4769-841e-3786c008e060 still found in kube manager 2025-12-22 01:40:50,877 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=d5e721e1-34be-4769-841e-3786c008e060 2025-12-22 01:40:50,891 - INFO - Ingress ctest-nginx-ingress-10459662 with uuid d5e721e1-34be-4769-841e-3786c008e060 deleted successfully from kube manager 2025-12-22 01:40:50,891 - INFO - Verifications on Ingress %s cleanup passed 2025-12-22 01:40:50,891 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-12-22 01:40:51,010 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-12-22 01:40:51,027 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-12-22 01:40:51,027 - INFO - Deleting VN __public__ 2025-12-22 01:40:51,248 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/5b3ed7a9-831b-448f-9bc0-7c7498c2e02b 2025-12-22 01:40:51,256 - DEBUG - Response Code: 404 2025-12-22 01:40:51,256 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2025-12-22 01:40:51,261 - DEBUG - Requesting: http://10.0.0.249:8082/domain/382747db-e7b4-4bbc-b2c8-0d9c92957c0e 2025-12-22 01:40:51,280 - DEBUG - Requesting: http://10.0.0.249:8082/project/6a507901-d7f1-454b-aeff-0cb7f1afa9ed 2025-12-22 01:40:51,359 - INFO - Validated that VN __public__ is not found in API Server 2025-12-22 01:40:51,359 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-12-22 01:40:51,364 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-22 01:40:51,369 - DEBUG - VN __public__ is not present in Agent 10.0.0.249 2025-12-22 01:40:51,369 - INFO - Validated that VN __public__ is not in any agent 2025-12-22 01:40:51,369 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-22 01:40:51,374 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-12-22 01:40:51,380 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-12-22 01:40:51,390 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-12-22 01:40:51,390 - INFO - Deleting pod default:ctest-nginx-pod-63079020 2025-12-22 01:40:51,403 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=1c1752cb-4ee6-41fb-8cbc-e4a6843cc2bf 2025-12-22 01:40:51,413 - WARNING - Pod uuid 1c1752cb-4ee6-41fb-8cbc-e4a6843cc2bf is still seen in agent 10.0.0.249 VM list 2025-12-22 01:40:56,414 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=1c1752cb-4ee6-41fb-8cbc-e4a6843cc2bf 2025-12-22 01:40:56,420 - DEBUG - Pod 1c1752cb-4ee6-41fb-8cbc-e4a6843cc2bf is not in agent 10.0.0.249 VM list 2025-12-22 01:40:56,420 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1c25db80-ded7-11f0-b5dc-024248d2bb15 2025-12-22 01:40:56,426 - DEBUG - VMI 1c25db80-ded7-11f0-b5dc-024248d2bb15 is removed from agent 10.0.0.249 2025-12-22 01:40:56,426 - INFO - Verified that pod ctest-nginx-pod-63079020 is removed in agent 2025-12-22 01:40:56,426 - INFO - Deleting pod default:ctest-nginx-pod-70124969 2025-12-22 01:40:56,439 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=256e6dad-3264-4057-a876-574cd03ede02 2025-12-22 01:40:56,448 - WARNING - Pod uuid 256e6dad-3264-4057-a876-574cd03ede02 is still seen in agent 10.0.0.249 VM list 2025-12-22 01:41:01,449 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=256e6dad-3264-4057-a876-574cd03ede02 2025-12-22 01:41:01,457 - DEBUG - Pod 256e6dad-3264-4057-a876-574cd03ede02 is not in agent 10.0.0.249 VM list 2025-12-22 01:41:01,457 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1b7377a6-ded7-11f0-b5dc-024248d2bb15 2025-12-22 01:41:01,469 - DEBUG - VMI 1b7377a6-ded7-11f0-b5dc-024248d2bb15 is removed from agent 10.0.0.249 2025-12-22 01:41:01,469 - INFO - Verified that pod ctest-nginx-pod-70124969 is removed in agent 2025-12-22 01:41:01,469 - INFO - Deleting service : ctest-nginx-svc-33420566 2025-12-22 01:41:02,082 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq? 2025-12-22 01:41:02,089 - DEBUG - No XMPP flaps were noticed during the test