2026-05-04 01:54:25,075 - DEBUG - [10.0.0.38]: Running cmd : hostname 2026-05-04 01:54:25,350 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-406-1 2026-05-04 01:54:25,350 - DEBUG - [10.0.0.38]: Running cmd : hostname -f 2026-05-04 01:54:25,470 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local 2026-05-04 01:54:25,470 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-05-04 01:54:25,643 - DEBUG - Output : NAMES contrail_test_tX3229DR9 k8s_autoscaler_dns-autoscaler-595558c478-sfgbk_kube-system_78816bb1-cfac-44a7-83e2-d7d177e9988c_0 k8s_coredns_coredns-77f7cc69db-m4p77_kube-system_0e88a682-5395-4e91-a1ca-b73f649f85b3_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-spszc_kube-system_75415091-ee0d-4c2f-ae5e-fa4e60766ca4_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-spszc_kube-system_75415091-ee0d-4c2f-ae5e-fa4e60766ca4_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-spszc_kube-system_75415091-ee0d-4c2f-ae5e-fa4e60766ca4_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-62nnq_kube-system_2c8dd0b3-11ee-4dd7-b2b8-2e1eff37e294_0 k8s_opensdn-controller-control-named_opensdn-controller-control-62nnq_kube-system_2c8dd0b3-11ee-4dd7-b2b8-2e1eff37e294_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-78g7b_kube-system_fe7be12e-46b8-45dc-9815-89ded81ef6bb_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-bsn8g_kube-system_a90cbc47-3369-44a6-92b9-05ec0d9a0e67_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-bsn8g_kube-system_a90cbc47-3369-44a6-92b9-05ec0d9a0e67_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-spszc_kube-system_75415091-ee0d-4c2f-ae5e-fa4e60766ca4_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-8kcrn_kube-system_2541e8a7-9a87-45a8-923b-09b8ad96622c_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-62nnq_kube-system_2c8dd0b3-11ee-4dd7-b2b8-2e1eff37e294_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-nfb6p_kube-system_9919284c-8ae8-44d7-9ae7-c51a6d25cf18_0 k8s_opensdn-analytics-collector_opensdn-analytics-nfb6p_kube-system_9919284c-8ae8-44d7-9ae7-c51a6d25cf18_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-wjtgl_kube-system_9fc5babb-1bcd-4ae9-9b6b-141f7a999f95_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-wjtgl_kube-system_9fc5babb-1bcd-4ae9-9b6b-141f7a999f95_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-78g7b_kube-system_fe7be12e-46b8-45dc-9815-89ded81ef6bb_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-bsn8g_kube-system_a90cbc47-3369-44a6-92b9-05ec0d9a0e67_0 k8s_opensdn-controller-config-api_opensdn-controller-config-spszc_kube-system_75415091-ee0d-4c2f-ae5e-fa4e60766ca4_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-8kcrn_kube-system_2541e8a7-9a87-45a8-923b-09b8ad96622c_0 k8s_opensdn-controller-control_opensdn-controller-control-62nnq_kube-system_2c8dd0b3-11ee-4dd7-b2b8-2e1eff37e294_0 k8s_opensdn-analytics-api_opensdn-analytics-nfb6p_kube-system_9919284c-8ae8-44d7-9ae7-c51a6d25cf18_0 k8s_kafka_opensdn-analytics-alarm-wjtgl_kube-system_9fc5babb-1bcd-4ae9-9b6b-141f7a999f95_0 k8s_config-zookeeper_config-zookeeper-xph26_kube-system_ce2afc67-4761-4834-bffc-0c020aab6b8d_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-xtsgl_kube-system_262e8ed7-be9b-41ae-ac57-4878411b5900_0 k8s_opensdn-configdb_opensdn-configdb-xtsgl_kube-system_262e8ed7-be9b-41ae-ac57-4878411b5900_0 k8s_rabbitmq_rabbitmq-whkpb_kube-system_d782dd6d-8c53-43ca-9a36-785b996759c1_0 k8s_redis_redis-mcnqq_kube-system_6b84e62b-7ed5-4d18-9bdc-3aa03df8c098_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-78g7b_kube-system_fe7be12e-46b8-45dc-9815-89ded81ef6bb_0 k8s_opensdn-agent-nodemgr_opensdn-agent-bb7j5_kube-system_7942bfe2-e21b-440b-8359-3d1893ee9297_0 k8s_opensdn-vrouter-agent_opensdn-agent-bb7j5_kube-system_7942bfe2-e21b-440b-8359-3d1893ee9297_0 k8s_opensdn-kube-manager_opensdn-kube-manager-gbkmt_kube-system_7e6cfcec-6076-47c5-ad45-077f059eb3ee_0 k8s_kube-proxy_kube-proxy-74ndd_kube-system_8b29e27a-02ea-4eea-b586-7299f38c329c_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-406-1_kube-system_9cb3692ac1dde934b56ee654f9051358_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-406-1_kube-system_59d0390221bc2605b3955786ec5fdd45_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-406-1_kube-system_891d9836d861a7659a6cc954f8b06a18_0 2026-05-04 01:54:25,644 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-05-04 01:54:25,770 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.38/24 2026-05-04 01:54:25,770 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}' 2026-05-04 01:54:25,894 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-406-1.cluster.local 2026-05-04 01:54:25,894 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-05-04 01:54:26,017 - DEBUG - Output : 10.0.0.38/24 2026-05-04 01:54:26,423 - INFO - Using existing project ['default-domain', 'k8s-default'](393d083c-74c5-4380-a33d-cb1c3725f91e) 2026-05-04 01:54:26,572 - INFO - Using existing project ['default-domain', 'k8s-default'](393d083c-74c5-4380-a33d-cb1c3725f91e) 2026-05-04 01:54:26,699 - INFO - ================================================================================ 2026-05-04 01:54:26,700 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-05-04 01:54:26,700 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-05-04 01:54:27,015 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_AgentXmppConnectionStatusReq? 2026-05-04 01:54:27,022 - DEBUG - Nothing to compare xmpp stats {'10.0.0.38': {'10.0.0.38': '2'}} with 2026-05-04 01:54:27,022 - INFO - Initial checks done. Running the testcase now 2026-05-04 01:54:27,022 - INFO - 2026-05-04 01:54:27,027 - INFO - Namespace default already exists 2026-05-04 01:54:27,027 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-gbkmt_kube-system_7e6cfcec-6076-47c5-ad45-077f059eb3ee_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-05-04 01:54:27,232 - DEBUG - Output : cluster_project={} 2026-05-04 01:54:27,323 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-05-04 01:54:27,323 - DEBUG - Project uuid in Contrail is 393d083c-74c5-4380-a33d-cb1c3725f91e 2026-05-04 01:54:27,323 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-gbkmt_kube-system_7e6cfcec-6076-47c5-ad45-077f059eb3ee_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-05-04 01:54:27,484 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-05-04 01:54:27,484 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_NamespaceDatabaseList?namespace_uuid=673f1ee4-e69b-49b1-ae40-b70e0e8eff37 2026-05-04 01:54:27,490 - INFO - Namespace default with uuid 673f1ee4-e69b-49b1-ae40-b70e0e8eff37 found in kube manager 2026-05-04 01:54:27,490 - INFO - Namespace default verification passed 2026-05-04 01:54:27,495 - DEBUG - Service ctest-nginx-svc-52231913 not present 2026-05-04 01:54:27,495 - INFO - Creating service ctest-nginx-svc-52231913 2026-05-04 01:54:27,511 - INFO - Created Service ctest-nginx-svc-52231913 2026-05-04 01:54:27,516 - DEBUG - Pod ctest-nginx-pod-84377968 not present 2026-05-04 01:54:27,517 - INFO - Creating Pod ctest-nginx-pod-84377968 2026-05-04 01:54:27,544 - DEBUG - Pod : ctest-nginx-pod-84377968 UUID is 8ea7b823-ef18-4b40-ad06-e317924258ec 2026-05-04 01:54:27,549 - DEBUG - Pod ctest-nginx-pod-00971284 not present 2026-05-04 01:54:27,550 - INFO - Creating Pod ctest-nginx-pod-00971284 2026-05-04 01:54:27,575 - DEBUG - Pod : ctest-nginx-pod-00971284 UUID is 00abe3a8-6d8a-4631-93fb-ddab562ef615 2026-05-04 01:54:27,918 - INFO - Created VN __public__, UUID :17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:28,147 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-05-04 01:54:28,156 - DEBUG - Requesting: http://10.0.0.38:8082/domain/735ea02e-8878-47bd-8316-674468c54a83 2026-05-04 01:54:28,171 - DEBUG - Requesting: http://10.0.0.38:8082/project/393d083c-74c5-4380-a33d-cb1c3725f91e 2026-05-04 01:54:28,239 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:28,250 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:28,265 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/306f231f-302f-472e-843c-a8bcf2a27bbc 2026-05-04 01:54:28,285 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/306f231f-302f-472e-843c-a8bcf2a27bbc 2026-05-04 01:54:28,292 - DEBUG - Route Targets: [] 2026-05-04 01:54:28,292 - DEBUG - RT names not yet present for VN __public__ 2026-05-04 01:54:33,292 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-05-04 01:54:33,302 - DEBUG - Requesting: http://10.0.0.38:8082/domain/735ea02e-8878-47bd-8316-674468c54a83 2026-05-04 01:54:33,314 - DEBUG - Requesting: http://10.0.0.38:8082/project/393d083c-74c5-4380-a33d-cb1c3725f91e 2026-05-04 01:54:33,372 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:33,380 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:33,387 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/306f231f-302f-472e-843c-a8bcf2a27bbc 2026-05-04 01:54:33,394 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/306f231f-302f-472e-843c-a8bcf2a27bbc 2026-05-04 01:54:33,400 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/4ed49eab-8e13-4fd8-8512-0bd5b25a1510 2026-05-04 01:54:33,405 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-05-04 01:54:33,405 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:33,411 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/306f231f-302f-472e-843c-a8bcf2a27bbc 2026-05-04 01:54:33,422 - INFO - Verified VN network id 11 for VN 17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:33,423 - INFO - Verifications in API Server for VN __public__ passed 2026-05-04 01:54:33,424 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:33,432 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/306f231f-302f-472e-843c-a8bcf2a27bbc 2026-05-04 01:54:33,438 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/306f231f-302f-472e-843c-a8bcf2a27bbc 2026-05-04 01:54:33,444 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/4ed49eab-8e13-4fd8-8512-0bd5b25a1510 2026-05-04 01:54:33,450 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-05-04 01:54:33,459 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1717042404587487288', 'uuid-lslong': '12896941664241171558'}, 'enable': 'true', 'created': '2026-05-04T01:54:27', 'last-modified': '2026-05-04T01:54:28', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.422222'} 2026-05-04 01:54:33,459 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-04 01:54:33,468 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-05-04 01:54:33,468 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-05-04 01:54:33,469 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-05-04 01:54:33,469 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-05-04 01:54:33,475 - DEBUG - Requesting: http://10.0.0.38:8082/domain/735ea02e-8878-47bd-8316-674468c54a83 2026-05-04 01:54:33,482 - DEBUG - Requesting: http://10.0.0.38:8082/project/393d083c-74c5-4380-a33d-cb1c3725f91e 2026-05-04 01:54:33,515 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:33,527 - DEBUG - =>VN __public__ has no policy to be verified 2026-05-04 01:54:33,527 - DEBUG - Verifying the vn in opserver 2026-05-04 01:54:33,528 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.38 2026-05-04 01:54:33,528 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2026-05-04 01:54:33,533 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-05-04 01:54:33,533 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-05-04 01:54:33,533 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-04 01:54:33,542 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-04 01:54:33,546 - DEBUG - Do not have enough data to verify VN in agent 2026-05-04 01:54:33,547 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-04 01:54:33,551 - DEBUG - VRF ids for VN __public__: {} 2026-05-04 01:54:33,742 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-05-04 01:54:33,956 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-05-04 01:54:33,961 - DEBUG - Requesting: http://10.0.0.38:8082/domain/735ea02e-8878-47bd-8316-674468c54a83 2026-05-04 01:54:33,970 - DEBUG - Requesting: http://10.0.0.38:8082/project/393d083c-74c5-4380-a33d-cb1c3725f91e 2026-05-04 01:54:34,069 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:34,092 - DEBUG - Requesting: http://10.0.0.38:8082/floating-ip-pool/8ac15eaa-ab3e-4eb7-964e-97c09a377a49 2026-05-04 01:54:34,105 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-05-04 01:54:34,117 - DEBUG - Requesting: http://10.0.0.38:8082/domain/735ea02e-8878-47bd-8316-674468c54a83 2026-05-04 01:54:34,144 - DEBUG - Requesting: http://10.0.0.38:8082/project/393d083c-74c5-4380-a33d-cb1c3725f91e 2026-05-04 01:54:34,283 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/17d428f8-3db8-4038-b2fb-241cda07bc66 2026-05-04 01:54:34,309 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-05-04 01:54:34,309 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-05-04 01:54:34,332 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-05-04 01:54:34,349 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-05-04 01:54:34,349 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-05-04 01:54:34,353 - DEBUG - Ingress ctest-nginx-ingress-63019086 not present 2026-05-04 01:54:34,354 - INFO - Creating Ingress ctest-nginx-ingress-63019086 2026-05-04 01:54:34,369 - INFO - Created Ingress ctest-nginx-ingress-63019086 2026-05-04 01:54:34,377 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-63019086 2026-05-04 01:54:37,384 - DEBUG - For Ingress ctest-nginx-ingress-63019086, Cluster IP: 10.47.255.248, External IPs ['22.72.201.124'] 2026-05-04 01:54:37,384 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-63019086 2026-05-04 01:54:37,384 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=35d1236f-57d8-4f59-ad55-ebfefe711639 2026-05-04 01:54:37,411 - INFO - Ingress ctest-nginx-ingress-63019086 with uuid 35d1236f-57d8-4f59-ad55-ebfefe711639 found in kube manager 2026-05-04 01:54:37,411 - INFO - Ingress ctest-nginx-ingress-63019086 verification passed 2026-05-04 01:54:37,416 - DEBUG - Pod ctest-busybox-pod-01271170 not present 2026-05-04 01:54:37,417 - INFO - Creating Pod ctest-busybox-pod-01271170 2026-05-04 01:54:37,448 - DEBUG - Pod : ctest-busybox-pod-01271170 UUID is f65e3dd5-c254-4839-a2c7-d29aeca3893e 2026-05-04 01:54:37,456 - DEBUG - Pod ctest-nginx-pod-84377968 not in running state.Currently in Pending 2026-05-04 01:54:42,469 - DEBUG - Pod ctest-nginx-pod-84377968 not in running state.Currently in Pending 2026-05-04 01:54:47,478 - INFO - Pod ctest-nginx-pod-84377968 is in running state.Got IP 10.47.255.250 2026-05-04 01:54:47,532 - DEBUG - Pod ctest-nginx-pod-84377968 has vmi 30879efe-475c-11f1-8ea1-0242a053f880 2026-05-04 01:54:47,532 - INFO - Verified pod ctest-nginx-pod-84377968 in contrail-api 2026-05-04 01:54:47,532 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-05-04 01:54:47,544 - DEBUG - VMI 30879efe-475c-11f1-8ea1-0242a053f880 is active in agent 10.0.0.38 2026-05-04 01:54:47,544 - INFO - Verified Pod ctest-nginx-pod-84377968 in agent 10.0.0.38 2026-05-04 01:54:47,545 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=8ea7b823-ef18-4b40-ad06-e317924258ec 2026-05-04 01:54:47,549 - INFO - Pod ctest-nginx-pod-84377968 with uuid 8ea7b823-ef18-4b40-ad06-e317924258ec found in kube manager 2026-05-04 01:54:47,549 - INFO - Pod ctest-nginx-pod-84377968 verification passed 2026-05-04 01:54:47,676 - DEBUG - [Pod ctest-nginx-pod-84377968] Cmd: echo ctest-nginx-pod-84377968 > /usr/share/nginx/html/index.html, Output: 2026-05-04 01:54:47,684 - INFO - Pod ctest-nginx-pod-00971284 is in running state.Got IP 10.47.255.249 2026-05-04 01:54:47,768 - DEBUG - Pod ctest-nginx-pod-00971284 has vmi 31384c4a-475c-11f1-8ea1-0242a053f880 2026-05-04 01:54:47,768 - INFO - Verified pod ctest-nginx-pod-00971284 in contrail-api 2026-05-04 01:54:47,768 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-05-04 01:54:47,777 - DEBUG - VMI 31384c4a-475c-11f1-8ea1-0242a053f880 is active in agent 10.0.0.38 2026-05-04 01:54:47,777 - INFO - Verified Pod ctest-nginx-pod-00971284 in agent 10.0.0.38 2026-05-04 01:54:47,777 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=00abe3a8-6d8a-4631-93fb-ddab562ef615 2026-05-04 01:54:47,781 - INFO - Pod ctest-nginx-pod-00971284 with uuid 00abe3a8-6d8a-4631-93fb-ddab562ef615 found in kube manager 2026-05-04 01:54:47,782 - INFO - Pod ctest-nginx-pod-00971284 verification passed 2026-05-04 01:54:47,877 - DEBUG - [Pod ctest-nginx-pod-00971284] Cmd: echo ctest-nginx-pod-00971284 > /usr/share/nginx/html/index.html, Output: 2026-05-04 01:54:47,884 - DEBUG - Pod ctest-busybox-pod-01271170 not in running state.Currently in Pending 2026-05-04 01:54:52,893 - DEBUG - Pod ctest-busybox-pod-01271170 not in running state.Currently in Pending 2026-05-04 01:54:57,906 - INFO - Pod ctest-busybox-pod-01271170 is in running state.Got IP 10.47.255.247 2026-05-04 01:54:57,965 - DEBUG - Pod ctest-busybox-pod-01271170 has vmi 368e5176-475c-11f1-8ea1-0242a053f880 2026-05-04 01:54:57,965 - INFO - Verified pod ctest-busybox-pod-01271170 in contrail-api 2026-05-04 01:54:57,966 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-05-04 01:54:57,975 - DEBUG - VMI 368e5176-475c-11f1-8ea1-0242a053f880 is active in agent 10.0.0.38 2026-05-04 01:54:57,976 - INFO - Verified Pod ctest-busybox-pod-01271170 in agent 10.0.0.38 2026-05-04 01:54:57,976 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=f65e3dd5-c254-4839-a2c7-d29aeca3893e 2026-05-04 01:54:57,980 - INFO - Pod ctest-busybox-pod-01271170 with uuid f65e3dd5-c254-4839-a2c7-d29aeca3893e found in kube manager 2026-05-04 01:54:57,981 - INFO - Pod ctest-busybox-pod-01271170 verification passed 2026-05-04 01:54:58,069 - DEBUG - [Pod ctest-busybox-pod-01271170] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-84377968 2026-05-04 01:54:58,069 - DEBUG - [Pod Pod ctest-busybox-pod-01271170] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-05-04 01:54:58,069 - DEBUG - [Pod Pod ctest-busybox-pod-01271170] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-84377968 2026-05-04 01:54:58,164 - DEBUG - [Pod ctest-busybox-pod-01271170] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-84377968 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-04 01:54:58,164 - DEBUG - [Pod Pod ctest-busybox-pod-01271170] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-05-04 01:54:58,164 - DEBUG - [Pod Pod ctest-busybox-pod-01271170] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-84377968 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-04 01:54:58,240 - DEBUG - [Pod ctest-busybox-pod-01271170] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-84377968 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-04 01:54:58,240 - DEBUG - [Pod Pod ctest-busybox-pod-01271170] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-05-04 01:54:58,240 - DEBUG - [Pod Pod ctest-busybox-pod-01271170] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-84377968 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-04 01:54:58,317 - DEBUG - [Pod ctest-busybox-pod-01271170] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-84377968 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-04 01:54:58,317 - DEBUG - [Pod Pod ctest-busybox-pod-01271170] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-05-04 01:54:58,317 - DEBUG - [Pod Pod ctest-busybox-pod-01271170] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-84377968 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-04 01:54:58,384 - DEBUG - [Pod ctest-busybox-pod-01271170] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-00971284 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-04 01:54:58,384 - DEBUG - [Pod Pod ctest-busybox-pod-01271170] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-05-04 01:54:58,384 - DEBUG - [Pod Pod ctest-busybox-pod-01271170] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-00971284 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-05-04 01:54:58,384 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-84377968': 4, 'ctest-nginx-pod-00971284': 1} 2026-05-04 01:54:58,385 - WARNING - Nothing to delete parallely 2026-05-04 01:54:58,385 - INFO - Deleting pod default:ctest-busybox-pod-01271170 2026-05-04 01:54:58,402 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f65e3dd5-c254-4839-a2c7-d29aeca3893e 2026-05-04 01:54:58,414 - WARNING - Pod uuid f65e3dd5-c254-4839-a2c7-d29aeca3893e is still seen in agent 10.0.0.38 VM list 2026-05-04 01:55:03,415 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f65e3dd5-c254-4839-a2c7-d29aeca3893e 2026-05-04 01:55:03,421 - WARNING - Pod uuid f65e3dd5-c254-4839-a2c7-d29aeca3893e is still seen in agent 10.0.0.38 VM list 2026-05-04 01:55:08,422 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f65e3dd5-c254-4839-a2c7-d29aeca3893e 2026-05-04 01:55:08,429 - WARNING - Pod uuid f65e3dd5-c254-4839-a2c7-d29aeca3893e is still seen in agent 10.0.0.38 VM list 2026-05-04 01:55:13,430 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f65e3dd5-c254-4839-a2c7-d29aeca3893e 2026-05-04 01:55:13,435 - WARNING - Pod uuid f65e3dd5-c254-4839-a2c7-d29aeca3893e is still seen in agent 10.0.0.38 VM list 2026-05-04 01:55:18,437 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f65e3dd5-c254-4839-a2c7-d29aeca3893e 2026-05-04 01:55:18,444 - WARNING - Pod uuid f65e3dd5-c254-4839-a2c7-d29aeca3893e is still seen in agent 10.0.0.38 VM list 2026-05-04 01:55:23,445 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f65e3dd5-c254-4839-a2c7-d29aeca3893e 2026-05-04 01:55:23,452 - WARNING - Pod uuid f65e3dd5-c254-4839-a2c7-d29aeca3893e is still seen in agent 10.0.0.38 VM list 2026-05-04 01:55:28,453 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f65e3dd5-c254-4839-a2c7-d29aeca3893e 2026-05-04 01:55:28,459 - WARNING - Pod uuid f65e3dd5-c254-4839-a2c7-d29aeca3893e is still seen in agent 10.0.0.38 VM list 2026-05-04 01:55:33,461 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f65e3dd5-c254-4839-a2c7-d29aeca3893e 2026-05-04 01:55:33,466 - DEBUG - Pod f65e3dd5-c254-4839-a2c7-d29aeca3893e is not in agent 10.0.0.38 VM list 2026-05-04 01:55:33,466 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:368e5176-475c-11f1-8ea1-0242a053f880 2026-05-04 01:55:33,476 - DEBUG - VMI 368e5176-475c-11f1-8ea1-0242a053f880 is removed from agent 10.0.0.38 2026-05-04 01:55:33,476 - INFO - Verified that pod ctest-busybox-pod-01271170 is removed in agent 2026-05-04 01:55:33,477 - INFO - Deleting Ingress : ctest-nginx-ingress-63019086 2026-05-04 01:55:33,487 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=35d1236f-57d8-4f59-ad55-ebfefe711639 2026-05-04 01:55:33,492 - ERROR - Ingress ctest-nginx-ingress-63019086 with uuid 35d1236f-57d8-4f59-ad55-ebfefe711639 still found in kube manager 2026-05-04 01:55:34,492 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=35d1236f-57d8-4f59-ad55-ebfefe711639 2026-05-04 01:55:34,503 - ERROR - Ingress ctest-nginx-ingress-63019086 with uuid 35d1236f-57d8-4f59-ad55-ebfefe711639 still found in kube manager 2026-05-04 01:55:35,503 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=35d1236f-57d8-4f59-ad55-ebfefe711639 2026-05-04 01:55:35,509 - INFO - Ingress ctest-nginx-ingress-63019086 with uuid 35d1236f-57d8-4f59-ad55-ebfefe711639 deleted successfully from kube manager 2026-05-04 01:55:35,509 - INFO - Verifications on Ingress %s cleanup passed 2026-05-04 01:55:35,509 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-05-04 01:55:35,604 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-05-04 01:55:35,610 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-05-04 01:55:35,610 - INFO - Deleting VN __public__ 2026-05-04 01:55:35,726 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/306f231f-302f-472e-843c-a8bcf2a27bbc 2026-05-04 01:55:35,742 - DEBUG - Response Code: 404 2026-05-04 01:55:35,742 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-05-04 01:55:35,751 - DEBUG - Requesting: http://10.0.0.38:8082/domain/735ea02e-8878-47bd-8316-674468c54a83 2026-05-04 01:55:35,764 - DEBUG - Requesting: http://10.0.0.38:8082/project/393d083c-74c5-4380-a33d-cb1c3725f91e 2026-05-04 01:55:35,797 - INFO - Validated that VN __public__ is not found in API Server 2026-05-04 01:55:35,797 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-05-04 01:55:35,805 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-04 01:55:35,813 - DEBUG - VN __public__ is not present in Agent 10.0.0.38 2026-05-04 01:55:35,813 - INFO - Validated that VN __public__ is not in any agent 2026-05-04 01:55:35,813 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-04 01:55:35,818 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-05-04 01:55:35,824 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-05-04 01:55:35,835 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-05-04 01:55:35,836 - INFO - Deleting pod default:ctest-nginx-pod-00971284 2026-05-04 01:55:35,852 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=00abe3a8-6d8a-4631-93fb-ddab562ef615 2026-05-04 01:55:35,861 - WARNING - Pod uuid 00abe3a8-6d8a-4631-93fb-ddab562ef615 is still seen in agent 10.0.0.38 VM list 2026-05-04 01:55:40,861 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=00abe3a8-6d8a-4631-93fb-ddab562ef615 2026-05-04 01:55:40,868 - DEBUG - Pod 00abe3a8-6d8a-4631-93fb-ddab562ef615 is not in agent 10.0.0.38 VM list 2026-05-04 01:55:40,868 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:31384c4a-475c-11f1-8ea1-0242a053f880 2026-05-04 01:55:40,874 - DEBUG - VMI 31384c4a-475c-11f1-8ea1-0242a053f880 is removed from agent 10.0.0.38 2026-05-04 01:55:40,874 - INFO - Verified that pod ctest-nginx-pod-00971284 is removed in agent 2026-05-04 01:55:40,874 - INFO - Deleting pod default:ctest-nginx-pod-84377968 2026-05-04 01:55:40,889 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=8ea7b823-ef18-4b40-ad06-e317924258ec 2026-05-04 01:55:40,895 - WARNING - Pod uuid 8ea7b823-ef18-4b40-ad06-e317924258ec is still seen in agent 10.0.0.38 VM list 2026-05-04 01:55:45,896 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=8ea7b823-ef18-4b40-ad06-e317924258ec 2026-05-04 01:55:45,902 - DEBUG - Pod 8ea7b823-ef18-4b40-ad06-e317924258ec is not in agent 10.0.0.38 VM list 2026-05-04 01:55:45,902 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:30879efe-475c-11f1-8ea1-0242a053f880 2026-05-04 01:55:45,906 - DEBUG - VMI 30879efe-475c-11f1-8ea1-0242a053f880 is removed from agent 10.0.0.38 2026-05-04 01:55:45,906 - INFO - Verified that pod ctest-nginx-pod-84377968 is removed in agent 2026-05-04 01:55:45,907 - INFO - Deleting service : ctest-nginx-svc-52231913 2026-05-04 01:55:46,467 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_AgentXmppConnectionStatusReq? 2026-05-04 01:55:46,473 - DEBUG - No XMPP flaps were noticed during the test