2026-02-10 00:20:04,813 - INFO - Using existing project ['default-domain', 'k8s-default'](525c4468-3eec-4cc8-8274-20534aa2eb19) 2026-02-10 00:20:04,833 - DEBUG - Namespace ctest-new-default-93282662 not present 2026-02-10 00:20:04,849 - INFO - Created namespace ctest-new-default-93282662 2026-02-10 00:20:07,853 - DEBUG - Namespace ctest-non-default-44517828 not present 2026-02-10 00:20:07,860 - INFO - Created namespace ctest-non-default-44517828 2026-02-10 00:20:10,882 - DEBUG - Pod ctest-pod-16279501 not present 2026-02-10 00:20:10,883 - INFO - Creating Pod ctest-pod-16279501 2026-02-10 00:20:10,910 - DEBUG - Pod : ctest-pod-16279501 UUID is e9b9b529-da66-4c1e-ad33-9f0930fd3ba3 2026-02-10 00:20:10,913 - DEBUG - Pod ctest-pod-23299811 not present 2026-02-10 00:20:10,914 - INFO - Creating Pod ctest-pod-23299811 2026-02-10 00:20:10,958 - DEBUG - Pod : ctest-pod-23299811 UUID is 61c98340-9a8b-42f4-afb8-3f12a0f23db7 2026-02-10 00:20:10,960 - DEBUG - Pod ctest-pod-42672806 not present 2026-02-10 00:20:10,961 - INFO - Creating Pod ctest-pod-42672806 2026-02-10 00:20:10,987 - DEBUG - Pod : ctest-pod-42672806 UUID is 03fa1cd4-6efe-4136-aedc-2f305f98245f 2026-02-10 00:20:10,989 - DEBUG - Pod ctest-pod-29866951 not present 2026-02-10 00:20:10,990 - INFO - Creating Pod ctest-pod-29866951 2026-02-10 00:20:11,017 - DEBUG - Pod : ctest-pod-29866951 UUID is 56def208-924e-47fd-bd6b-9fd0cff1421b 2026-02-10 00:20:11,019 - DEBUG - Pod ctest-pod-64542976 not present 2026-02-10 00:20:11,019 - INFO - Creating Pod ctest-pod-64542976 2026-02-10 00:20:11,040 - DEBUG - Pod : ctest-pod-64542976 UUID is a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 2026-02-10 00:20:11,042 - DEBUG - Pod ctest-pod-25423383 not present 2026-02-10 00:20:11,042 - INFO - Creating Pod ctest-pod-25423383 2026-02-10 00:20:11,060 - DEBUG - Pod : ctest-pod-25423383 UUID is 3e9e4e6e-4235-4aca-a696-18764141527a 2026-02-10 00:20:11,062 - DEBUG - Pod ctest-pod-47696277 not present 2026-02-10 00:20:11,062 - INFO - Creating Pod ctest-pod-47696277 2026-02-10 00:20:11,096 - DEBUG - Pod : ctest-pod-47696277 UUID is 814a21f2-58b0-4a61-b6cd-b12356a829da 2026-02-10 00:20:11,098 - DEBUG - Pod ctest-pod-56223887 not present 2026-02-10 00:20:11,100 - INFO - Creating Pod ctest-pod-56223887 2026-02-10 00:20:11,127 - DEBUG - Pod : ctest-pod-56223887 UUID is 8bd7ea63-a75a-4638-b53f-130a77b850c8 2026-02-10 00:20:11,390 - INFO - Project default-domain:k8s-ctest-new-default-93282662 for Namespace ctest-new-default-93282662 is seen in contrail-api 2026-02-10 00:20:11,391 - DEBUG - Project uuid in Contrail is d011f8ef-de98-4b96-bc65-6da0c4b1d535 2026-02-10 00:20:11,551 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_NamespaceDatabaseList?namespace_uuid=cd0d3cd2-4199-4e60-949d-274084fd9793 2026-02-10 00:20:11,557 - INFO - Namespace ctest-new-default-93282662 with uuid cd0d3cd2-4199-4e60-949d-274084fd9793 found in kube manager 2026-02-10 00:20:11,557 - INFO - Namespace ctest-new-default-93282662 verification passed 2026-02-10 00:20:11,791 - INFO - Project default-domain:k8s-ctest-non-default-44517828 for Namespace ctest-non-default-44517828 is seen in contrail-api 2026-02-10 00:20:11,791 - DEBUG - Project uuid in Contrail is 3c8f350a-f250-484d-b1ce-45b07e637faf 2026-02-10 00:20:11,791 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_NamespaceDatabaseList?namespace_uuid=596166a5-2902-434e-887d-68aa60239eeb 2026-02-10 00:20:11,795 - INFO - Namespace ctest-non-default-44517828 with uuid 596166a5-2902-434e-887d-68aa60239eeb found in kube manager 2026-02-10 00:20:11,795 - INFO - Namespace ctest-non-default-44517828 verification passed 2026-02-10 00:20:11,801 - DEBUG - Pod ctest-pod-16279501 not in running state.Currently in Pending 2026-02-10 00:20:16,810 - DEBUG - Pod ctest-pod-16279501 not in running state.Currently in Pending 2026-02-10 00:20:21,821 - INFO - Pod ctest-pod-16279501 is in running state.Got IP 10.47.255.251 2026-02-10 00:20:21,833 - DEBUG - Pod ctest-pod-16279501 has vmi 43ffb55c-0616-11f1-bf55-fa163e6f1404 2026-02-10 00:20:21,833 - INFO - Verified pod ctest-pod-16279501 in contrail-api 2026-02-10 00:20:21,844 - DEBUG - VMI 43ffb55c-0616-11f1-bf55-fa163e6f1404 is active in agent 10.0.0.240 2026-02-10 00:20:21,844 - INFO - Verified Pod ctest-pod-16279501 in agent 10.0.0.240 2026-02-10 00:20:21,844 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_VirtualMachineDatabaseList?x=e9b9b529-da66-4c1e-ad33-9f0930fd3ba3 2026-02-10 00:20:21,848 - INFO - Pod ctest-pod-16279501 with uuid e9b9b529-da66-4c1e-ad33-9f0930fd3ba3 found in kube manager 2026-02-10 00:20:21,848 - INFO - Pod ctest-pod-16279501 verification passed 2026-02-10 00:20:21,854 - DEBUG - Pod ctest-pod-23299811 not in running state.Currently in Pending 2026-02-10 00:20:26,862 - INFO - Pod ctest-pod-23299811 is in running state.Got IP 10.47.255.250 2026-02-10 00:20:26,917 - DEBUG - Pod ctest-pod-23299811 has vmi 446cfe5a-0616-11f1-bf55-fa163e6f1404 2026-02-10 00:20:26,917 - INFO - Verified pod ctest-pod-23299811 in contrail-api 2026-02-10 00:20:26,929 - DEBUG - VMI 446cfe5a-0616-11f1-bf55-fa163e6f1404 is active in agent 10.0.0.240 2026-02-10 00:20:26,930 - INFO - Verified Pod ctest-pod-23299811 in agent 10.0.0.240 2026-02-10 00:20:26,930 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_VirtualMachineDatabaseList?x=61c98340-9a8b-42f4-afb8-3f12a0f23db7 2026-02-10 00:20:26,933 - INFO - Pod ctest-pod-23299811 with uuid 61c98340-9a8b-42f4-afb8-3f12a0f23db7 found in kube manager 2026-02-10 00:20:26,933 - INFO - Pod ctest-pod-23299811 verification passed 2026-02-10 00:20:26,939 - INFO - Pod ctest-pod-42672806 is in running state.Got IP 10.47.255.249 2026-02-10 00:20:26,955 - DEBUG - Pod ctest-pod-42672806 has vmi 44fbe07a-0616-11f1-bf55-fa163e6f1404 2026-02-10 00:20:26,955 - INFO - Verified pod ctest-pod-42672806 in contrail-api 2026-02-10 00:20:26,969 - DEBUG - VMI 44fbe07a-0616-11f1-bf55-fa163e6f1404 is active in agent 10.0.0.240 2026-02-10 00:20:26,970 - INFO - Verified Pod ctest-pod-42672806 in agent 10.0.0.240 2026-02-10 00:20:26,970 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_VirtualMachineDatabaseList?x=03fa1cd4-6efe-4136-aedc-2f305f98245f 2026-02-10 00:20:26,974 - INFO - Pod ctest-pod-42672806 with uuid 03fa1cd4-6efe-4136-aedc-2f305f98245f found in kube manager 2026-02-10 00:20:26,974 - INFO - Pod ctest-pod-42672806 verification passed 2026-02-10 00:20:26,980 - INFO - Pod ctest-pod-29866951 is in running state.Got IP 10.47.255.248 2026-02-10 00:20:27,036 - DEBUG - Pod ctest-pod-29866951 has vmi 4571562a-0616-11f1-bf55-fa163e6f1404 2026-02-10 00:20:27,037 - INFO - Verified pod ctest-pod-29866951 in contrail-api 2026-02-10 00:20:27,053 - DEBUG - VMI 4571562a-0616-11f1-bf55-fa163e6f1404 is active in agent 10.0.0.240 2026-02-10 00:20:27,053 - INFO - Verified Pod ctest-pod-29866951 in agent 10.0.0.240 2026-02-10 00:20:27,053 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_VirtualMachineDatabaseList?x=56def208-924e-47fd-bd6b-9fd0cff1421b 2026-02-10 00:20:27,060 - INFO - Pod ctest-pod-29866951 with uuid 56def208-924e-47fd-bd6b-9fd0cff1421b found in kube manager 2026-02-10 00:20:27,060 - INFO - Pod ctest-pod-29866951 verification passed 2026-02-10 00:20:27,068 - DEBUG - Pod ctest-pod-64542976 not in running state.Currently in Pending 2026-02-10 00:20:32,075 - INFO - Pod ctest-pod-64542976 is in running state.Got IP 10.47.255.247 2026-02-10 00:20:32,087 - DEBUG - Pod ctest-pod-64542976 has vmi 45f6091a-0616-11f1-bf55-fa163e6f1404 2026-02-10 00:20:32,087 - INFO - Verified pod ctest-pod-64542976 in contrail-api 2026-02-10 00:20:32,099 - DEBUG - VMI 45f6091a-0616-11f1-bf55-fa163e6f1404 is active in agent 10.0.0.240 2026-02-10 00:20:32,099 - INFO - Verified Pod ctest-pod-64542976 in agent 10.0.0.240 2026-02-10 00:20:32,099 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_VirtualMachineDatabaseList?x=a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 2026-02-10 00:20:32,105 - INFO - Pod ctest-pod-64542976 with uuid a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 found in kube manager 2026-02-10 00:20:32,105 - INFO - Pod ctest-pod-64542976 verification passed 2026-02-10 00:20:32,111 - INFO - Pod ctest-pod-47696277 is in running state.Got IP 10.47.255.245 2026-02-10 00:20:32,169 - DEBUG - Pod ctest-pod-47696277 has vmi 46c629ba-0616-11f1-bf55-fa163e6f1404 2026-02-10 00:20:32,169 - INFO - Verified pod ctest-pod-47696277 in contrail-api 2026-02-10 00:20:32,180 - DEBUG - VMI 46c629ba-0616-11f1-bf55-fa163e6f1404 is active in agent 10.0.0.240 2026-02-10 00:20:32,180 - INFO - Verified Pod ctest-pod-47696277 in agent 10.0.0.240 2026-02-10 00:20:32,180 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_VirtualMachineDatabaseList?x=814a21f2-58b0-4a61-b6cd-b12356a829da 2026-02-10 00:20:32,185 - INFO - Pod ctest-pod-47696277 with uuid 814a21f2-58b0-4a61-b6cd-b12356a829da found in kube manager 2026-02-10 00:20:32,185 - INFO - Pod ctest-pod-47696277 verification passed 2026-02-10 00:20:32,192 - INFO - Pod ctest-pod-25423383 is in running state.Got IP 10.47.255.246 2026-02-10 00:20:32,206 - DEBUG - Pod ctest-pod-25423383 has vmi 4660680a-0616-11f1-bf55-fa163e6f1404 2026-02-10 00:20:32,206 - INFO - Verified pod ctest-pod-25423383 in contrail-api 2026-02-10 00:20:32,217 - DEBUG - VMI 4660680a-0616-11f1-bf55-fa163e6f1404 is active in agent 10.0.0.240 2026-02-10 00:20:32,218 - INFO - Verified Pod ctest-pod-25423383 in agent 10.0.0.240 2026-02-10 00:20:32,218 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_VirtualMachineDatabaseList?x=3e9e4e6e-4235-4aca-a696-18764141527a 2026-02-10 00:20:32,221 - INFO - Pod ctest-pod-25423383 with uuid 3e9e4e6e-4235-4aca-a696-18764141527a found in kube manager 2026-02-10 00:20:32,221 - INFO - Pod ctest-pod-25423383 verification passed 2026-02-10 00:20:32,227 - INFO - Pod ctest-pod-56223887 is in running state.Got IP 10.47.255.244 2026-02-10 00:20:32,280 - DEBUG - Pod ctest-pod-56223887 has vmi 473614a0-0616-11f1-bf55-fa163e6f1404 2026-02-10 00:20:32,280 - INFO - Verified pod ctest-pod-56223887 in contrail-api 2026-02-10 00:20:32,299 - DEBUG - VMI 473614a0-0616-11f1-bf55-fa163e6f1404 is active in agent 10.0.0.240 2026-02-10 00:20:32,299 - INFO - Verified Pod ctest-pod-56223887 in agent 10.0.0.240 2026-02-10 00:20:32,300 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_VirtualMachineDatabaseList?x=8bd7ea63-a75a-4638-b53f-130a77b850c8 2026-02-10 00:20:32,304 - INFO - Pod ctest-pod-56223887 with uuid 8bd7ea63-a75a-4638-b53f-130a77b850c8 found in kube manager 2026-02-10 00:20:32,304 - INFO - Pod ctest-pod-56223887 verification passed 2026-02-10 00:20:32,304 - INFO - ================================================================================ 2026-02-10 00:20:32,304 - INFO - STARTING TEST : test_ingress_rule_on_pod_with_service 2026-02-10 00:20:32,305 - INFO - TEST DESCRIPTION : Verify that ingress rules on a pod work fine when pods are accessed via a service Steps: 1. Create a service in namespace 1 over a pod which are endpoints of a service 2. Create a network policy to allow 1 pod of namespace 1 as ingress. 3. Verify that service respond correctly to the specified pod in the policy and service do not respond to anyone else. 4. Extend the policy and add the namespaceselector as namespace 2. 5. Verify that service become accessible to the pods of namespace 2. 2026-02-10 00:20:32,508 - DEBUG - Nothing to compare xmpp stats {'10.0.0.240': {'10.0.0.23': '0'}} with 2026-02-10 00:20:32,508 - INFO - Initial checks done. Running the testcase now 2026-02-10 00:20:32,508 - INFO - 2026-02-10 00:20:32,562 - DEBUG - [Pod ctest-pod-16279501] Cmd: echo ctest-pod-16279501 > /usr/share/nginx/html/index.html, Output: 2026-02-10 00:20:32,618 - DEBUG - [Pod ctest-pod-23299811] Cmd: echo ctest-pod-23299811 > /usr/share/nginx/html/index.html, Output: 2026-02-10 00:20:32,682 - DEBUG - [Pod ctest-pod-42672806] Cmd: echo ctest-pod-42672806 > /usr/share/nginx/html/index.html, Output: 2026-02-10 00:20:32,735 - DEBUG - [Pod ctest-pod-29866951] Cmd: echo ctest-pod-29866951 > /usr/share/nginx/html/index.html, Output: 2026-02-10 00:20:32,738 - DEBUG - Service ctest-nginx-svc-64648844 not present 2026-02-10 00:20:32,738 - INFO - Creating service ctest-nginx-svc-64648844 2026-02-10 00:20:32,759 - INFO - Created Service ctest-nginx-svc-64648844 2026-02-10 00:20:32,772 - WARNING - Service UUID c9bfc914-82db-49f6-a593-48c9c41a83c2 not yet found in contrail-api 2026-02-10 00:20:33,789 - INFO - Validated that Service ctest-nginx-svc-64648844 is seen in contrail-api 2026-02-10 00:20:33,789 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_LoadbalancerDatabaseList?x=c9bfc914-82db-49f6-a593-48c9c41a83c2 2026-02-10 00:20:33,793 - INFO - Service ctest-nginx-svc-64648844 with uuid c9bfc914-82db-49f6-a593-48c9c41a83c2 found in kube manager 2026-02-10 00:20:33,793 - INFO - Service ctest-nginx-svc-64648844 verification passed 2026-02-10 00:20:34,861 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-23299811 2026-02-10 00:20:34,861 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 passed 2026-02-10 00:20:34,861 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-23299811 2026-02-10 00:20:34,916 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: ctest-pod-16279501 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:20:34,916 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 passed 2026-02-10 00:20:34,917 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: ctest-pod-16279501 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:20:34,917 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16279501': 1, 'ctest-pod-23299811': 1} 2026-02-10 00:20:34,976 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: ctest-pod-16279501 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:20:34,976 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 passed 2026-02-10 00:20:34,976 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: ctest-pod-16279501 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:20:35,033 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: ctest-pod-23299811 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:20:35,033 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 passed 2026-02-10 00:20:35,033 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: ctest-pod-23299811 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:20:35,033 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16279501': 1, 'ctest-pod-23299811': 1} 2026-02-10 00:20:35,036 - DEBUG - Network policy ingress-on-service not present 2026-02-10 00:20:35,037 - INFO - Creating Network Policy ingress-on-service 2026-02-10 00:20:35,052 - INFO - Network policy found in k8s 2026-02-10 00:20:35,052 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=06cce423-7e63-4600-9173-87bb0af180f2 2026-02-10 00:20:35,057 - INFO - Network Policy ingress-on-service with uuid 06cce423-7e63-4600-9173-87bb0af180f2 found in kube manager 2026-02-10 00:20:35,084 - INFO - Network policy with name default-policy-management:k8s-allowall found in agent 2026-02-10 00:20:35,084 - INFO - Network policy with name default-policy-management:k8s-allowall associated with default ks8 2026-02-10 00:20:35,085 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=06cce423-7e63-4600-9173-87bb0af180f2 2026-02-10 00:20:35,109 - WARNING - Network policy with name ingress-on-service not found in agent 2026-02-10 00:20:37,110 - DEBUG - Requesting: http://10.0.0.23:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=06cce423-7e63-4600-9173-87bb0af180f2 2026-02-10 00:20:37,133 - INFO - Network policy with name ingress-on-service found in agent 2026-02-10 00:20:37,133 - INFO - Network Policy ingress-on-service verification passed 2026-02-10 00:20:42,184 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:42,184 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:42,184 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:47,225 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:47,225 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:47,225 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:52,278 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:52,278 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:52,278 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:57,334 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:57,334 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:20:57,334 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:02,395 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:02,395 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:02,395 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:07,445 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:07,445 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:07,445 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:12,510 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:12,510 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:12,510 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:17,562 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:17,562 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:17,562 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:22,613 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:22,613 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:22,613 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:27,661 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:27,661 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:27,661 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:27,661 - INFO - As expected, responses not seen from podsHits : {'ctest-pod-16279501': 0, 'ctest-pod-23299811': 0} 2026-02-10 00:21:27,720 - DEBUG - [Pod ctest-pod-25423383] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: ctest-pod-16279501 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:21:27,720 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 passed 2026-02-10 00:21:27,720 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd output: ctest-pod-16279501 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:21:27,787 - DEBUG - [Pod ctest-pod-25423383] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: ctest-pod-16279501 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:21:27,787 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 passed 2026-02-10 00:21:27,787 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd output: ctest-pod-16279501 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:21:27,844 - DEBUG - [Pod ctest-pod-25423383] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-16279501 2026-02-10 00:21:27,844 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 passed 2026-02-10 00:21:27,844 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-16279501 2026-02-10 00:21:27,894 - DEBUG - [Pod ctest-pod-25423383] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: ctest-pod-23299811 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:21:27,894 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 passed 2026-02-10 00:21:27,894 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd output: ctest-pod-23299811 Connecting to 10.100.161.2:80 (10.100.161.2:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-10 00:21:27,894 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16279501': 3, 'ctest-pod-23299811': 1} 2026-02-10 00:21:32,933 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:32,933 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:32,933 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:37,994 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:37,994 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:37,994 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:39,693 - WARNING - ApiException is caught (0) Reason: Connection is already closed. . Retrying... 2026-02-10 00:21:49,753 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:49,753 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:49,753 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:54,804 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:54,805 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:54,805 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:59,854 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:59,861 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:21:59,861 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:04,930 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:04,930 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:04,930 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:09,992 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:09,993 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:09,993 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:15,041 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:15,042 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:15,042 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:20,095 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:20,095 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:20,095 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:25,143 - DEBUG - [Pod ctest-pod-47696277] Cmd: wget http://10.100.161.2:80/ -O - -T 5 -t 1 , Output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:25,143 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd wget http://10.100.161.2:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:25,143 - DEBUG - [Pod Pod ctest-pod-47696277] Cmd output: Connecting to 10.100.161.2:80 (10.100.161.2:80) wget: download timed out 2026-02-10 00:22:25,143 - INFO - As expected, responses not seen from podsHits : {'ctest-pod-16279501': 0, 'ctest-pod-23299811': 0} 2026-02-10 00:22:27,193 - DEBUG - [Pod ctest-pod-25423383] Cmd: ping -c 3 10.47.255.247, Output: PING 10.47.255.247 (10.47.255.247): 56 data bytes 64 bytes from 10.47.255.247: seq=0 ttl=63 time=0.539 ms 64 bytes from 10.47.255.247: seq=1 ttl=63 time=0.070 ms 64 bytes from 10.47.255.247: seq=2 ttl=63 time=0.078 ms --- 10.47.255.247 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.070/0.229/0.539 ms 2026-02-10 00:22:27,193 - INFO - Ping check to IP 10.47.255.247 from pod ctest-pod-25423383 with expectation True passed 2026-02-10 00:22:29,245 - DEBUG - [Pod ctest-pod-64542976] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes 64 bytes from 10.47.255.246: seq=0 ttl=63 time=1.145 ms 64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.069 ms 64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.098 ms --- 10.47.255.246 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.069/0.437/1.145 ms 2026-02-10 00:22:29,246 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-64542976 with expectation True passed 2026-02-10 00:22:31,298 - DEBUG - [Pod ctest-pod-47696277] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes 64 bytes from 10.47.255.246: seq=0 ttl=63 time=0.588 ms 64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.073 ms 64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.093 ms --- 10.47.255.246 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.073/0.251/0.588 ms 2026-02-10 00:22:31,298 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-47696277 with expectation True passed 2026-02-10 00:22:31,351 - DEBUG - [Pod ctest-pod-25423383] Cmd: wget http://10.47.255.251 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.251 (10.47.255.251:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:31,352 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd wget http://10.47.255.251 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:31,352 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd output: Connecting to 10.47.255.251 (10.47.255.251:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:31,352 - INFO - wget check of of http://10.47.255.251 from ctest-pod-25423383 passed 2026-02-10 00:22:31,407 - DEBUG - [Pod ctest-pod-25423383] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:31,407 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:31,407 - DEBUG - [Pod Pod ctest-pod-25423383] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:31,407 - INFO - wget check of of http://10.47.255.250 from ctest-pod-25423383 passed 2026-02-10 00:22:31,462 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:31,462 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:31,462 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:31,462 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:33,520 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:33,521 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:33,521 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:33,521 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:35,578 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:35,578 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:35,578 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:35,578 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:37,640 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:37,640 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:37,640 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:37,640 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:39,696 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:39,696 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:39,696 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:39,696 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:41,755 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:41,755 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:41,756 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:41,756 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:43,812 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:43,812 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:43,812 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:43,812 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:45,863 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:45,863 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:45,863 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:45,863 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:47,919 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:47,919 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:47,919 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:47,919 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:49,978 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:49,978 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:49,978 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:49,978 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:52,035 - DEBUG - [Pod ctest-pod-64542976] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:52,035 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-10 00:22:52,035 - DEBUG - [Pod Pod ctest-pod-64542976] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-10 00:22:52,036 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-64542976 failed 2026-02-10 00:22:52,040 - INFO - Deleting Network Policy : ingress-on-service 2026-02-10 00:22:52,041 - INFO - Deleting service : ctest-nginx-svc-64648844 2026-02-10 00:22:52,281 - ERROR - AssertionError Python 3.9.21: /usr/bin/python3 Tue Feb 10 00:22:52 2026 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/k8s_scripts/test_policy.py in test_ingress_rule_on_pod_with_service(self=) 2859 assert self.validate_wget(self.client2_pod_ns1, url1) 2860 assert self.validate_wget(self.client2_pod_ns1, url2) 2861 assert self.validate_wget(self.client1_pod_ns1, url2, expectation=False) 2862 assert self.validate_wget(self.client2_pod_ns2, url1, expectation=False) 2863 # Updating the policy and adding the namespace selector self = self.validate_wget = .deco_retry....ice[ci_k8s_sanity,k8s_sanity] id=0x7fd55cbcfaf0>> self.client1_pod_ns1 = url2 = 'http://10.47.255.250' expectation undefined AssertionError: __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = () with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/k8s_scripts/test_policy.py", line 2861, in test_ingress_rule_on_pod_with_service assert self.validate_wget(self.client1_pod_ns1, url2, expectation=False) AssertionError 2026-02-10 00:22:52,287 - DEBUG - No XMPP flaps were noticed during the test 2026-02-10 00:22:52,287 - INFO -
2026-02-10 00:22:52,287 - INFO - END TEST : test_ingress_rule_on_pod_with_service : FAILED[0:02:20]
2026-02-10 00:22:52,287 - INFO - -------------------------------------------------------------------------------- 2026-02-10 00:22:52,291 - INFO - Deleting pod ctest-new-default-93282662:ctest-pod-16279501 2026-02-10 00:22:52,292 - INFO - Deleting pod ctest-new-default-93282662:ctest-pod-23299811 2026-02-10 00:22:52,292 - INFO - Deleting pod ctest-non-default-44517828:ctest-pod-42672806 2026-02-10 00:22:52,293 - INFO - Deleting pod ctest-non-default-44517828:ctest-pod-29866951 2026-02-10 00:22:52,293 - INFO - Deleting pod ctest-new-default-93282662:ctest-pod-64542976 2026-02-10 00:22:52,293 - INFO - Deleting pod ctest-new-default-93282662:ctest-pod-25423383 2026-02-10 00:22:52,294 - INFO - Deleting pod ctest-non-default-44517828:ctest-pod-47696277 2026-02-10 00:22:52,294 - INFO - Deleting pod ctest-non-default-44517828:ctest-pod-56223887 2026-02-10 00:22:52,450 - WARNING - Pod uuid a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:52,450 - WARNING - Pod uuid e9b9b529-da66-4c1e-ad33-9f0930fd3ba3 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:52,451 - WARNING - Pod uuid 814a21f2-58b0-4a61-b6cd-b12356a829da is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:52,451 - WARNING - Pod uuid 03fa1cd4-6efe-4136-aedc-2f305f98245f is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:52,452 - WARNING - Pod uuid 8bd7ea63-a75a-4638-b53f-130a77b850c8 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:52,452 - WARNING - Pod uuid 61c98340-9a8b-42f4-afb8-3f12a0f23db7 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:52,452 - WARNING - Pod uuid 56def208-924e-47fd-bd6b-9fd0cff1421b is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:52,453 - WARNING - Pod uuid 3e9e4e6e-4235-4aca-a696-18764141527a is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:57,478 - WARNING - Pod uuid a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:57,496 - DEBUG - Pod 03fa1cd4-6efe-4136-aedc-2f305f98245f is not in agent 10.0.0.240 VM list 2026-02-10 00:22:57,497 - WARNING - Pod uuid 8bd7ea63-a75a-4638-b53f-130a77b850c8 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:57,498 - WARNING - Pod uuid 814a21f2-58b0-4a61-b6cd-b12356a829da is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:57,500 - WARNING - Pod uuid 3e9e4e6e-4235-4aca-a696-18764141527a is still seen in agent 10.0.0.240 VM list 2026-02-10 00:22:57,500 - DEBUG - Pod 56def208-924e-47fd-bd6b-9fd0cff1421b is not in agent 10.0.0.240 VM list 2026-02-10 00:22:57,503 - DEBUG - Pod 61c98340-9a8b-42f4-afb8-3f12a0f23db7 is not in agent 10.0.0.240 VM list 2026-02-10 00:22:57,504 - DEBUG - Pod e9b9b529-da66-4c1e-ad33-9f0930fd3ba3 is not in agent 10.0.0.240 VM list 2026-02-10 00:22:57,519 - DEBUG - VMI 43ffb55c-0616-11f1-bf55-fa163e6f1404 is removed from agent 10.0.0.240 2026-02-10 00:22:57,519 - INFO - Verified that pod ctest-pod-16279501 is removed in agent 2026-02-10 00:22:57,520 - DEBUG - VMI 446cfe5a-0616-11f1-bf55-fa163e6f1404 is removed from agent 10.0.0.240 2026-02-10 00:22:57,520 - INFO - Verified that pod ctest-pod-23299811 is removed in agent 2026-02-10 00:22:57,520 - DEBUG - VMI 4571562a-0616-11f1-bf55-fa163e6f1404 is removed from agent 10.0.0.240 2026-02-10 00:22:57,520 - INFO - Verified that pod ctest-pod-29866951 is removed in agent 2026-02-10 00:22:57,521 - DEBUG - VMI 44fbe07a-0616-11f1-bf55-fa163e6f1404 is removed from agent 10.0.0.240 2026-02-10 00:22:57,521 - INFO - Verified that pod ctest-pod-42672806 is removed in agent 2026-02-10 00:23:02,485 - WARNING - Pod uuid a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:02,509 - WARNING - Pod uuid 8bd7ea63-a75a-4638-b53f-130a77b850c8 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:02,510 - WARNING - Pod uuid 3e9e4e6e-4235-4aca-a696-18764141527a is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:02,511 - WARNING - Pod uuid 814a21f2-58b0-4a61-b6cd-b12356a829da is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:07,491 - WARNING - Pod uuid a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:07,521 - WARNING - Pod uuid 8bd7ea63-a75a-4638-b53f-130a77b850c8 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:07,523 - WARNING - Pod uuid 3e9e4e6e-4235-4aca-a696-18764141527a is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:07,523 - WARNING - Pod uuid 814a21f2-58b0-4a61-b6cd-b12356a829da is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:12,499 - WARNING - Pod uuid a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:12,535 - WARNING - Pod uuid 814a21f2-58b0-4a61-b6cd-b12356a829da is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:12,535 - WARNING - Pod uuid 8bd7ea63-a75a-4638-b53f-130a77b850c8 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:12,536 - WARNING - Pod uuid 3e9e4e6e-4235-4aca-a696-18764141527a is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:17,507 - WARNING - Pod uuid a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:17,547 - WARNING - Pod uuid 814a21f2-58b0-4a61-b6cd-b12356a829da is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:17,548 - WARNING - Pod uuid 8bd7ea63-a75a-4638-b53f-130a77b850c8 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:17,548 - WARNING - Pod uuid 3e9e4e6e-4235-4aca-a696-18764141527a is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:22,514 - WARNING - Pod uuid a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:22,559 - WARNING - Pod uuid 814a21f2-58b0-4a61-b6cd-b12356a829da is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:22,561 - WARNING - Pod uuid 3e9e4e6e-4235-4aca-a696-18764141527a is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:22,561 - WARNING - Pod uuid 8bd7ea63-a75a-4638-b53f-130a77b850c8 is still seen in agent 10.0.0.240 VM list 2026-02-10 00:23:27,522 - DEBUG - Pod a2f0e6e9-0a85-4f02-8de5-6ad45eca8993 is not in agent 10.0.0.240 VM list 2026-02-10 00:23:27,528 - DEBUG - VMI 45f6091a-0616-11f1-bf55-fa163e6f1404 is removed from agent 10.0.0.240 2026-02-10 00:23:27,528 - INFO - Verified that pod ctest-pod-64542976 is removed in agent 2026-02-10 00:23:27,571 - DEBUG - Pod 814a21f2-58b0-4a61-b6cd-b12356a829da is not in agent 10.0.0.240 VM list 2026-02-10 00:23:27,575 - DEBUG - Pod 8bd7ea63-a75a-4638-b53f-130a77b850c8 is not in agent 10.0.0.240 VM list 2026-02-10 00:23:27,576 - DEBUG - Pod 3e9e4e6e-4235-4aca-a696-18764141527a is not in agent 10.0.0.240 VM list 2026-02-10 00:23:27,585 - DEBUG - VMI 46c629ba-0616-11f1-bf55-fa163e6f1404 is removed from agent 10.0.0.240 2026-02-10 00:23:27,585 - INFO - Verified that pod ctest-pod-47696277 is removed in agent 2026-02-10 00:23:27,586 - DEBUG - VMI 473614a0-0616-11f1-bf55-fa163e6f1404 is removed from agent 10.0.0.240 2026-02-10 00:23:27,586 - INFO - Verified that pod ctest-pod-56223887 is removed in agent 2026-02-10 00:23:27,586 - DEBUG - VMI 4660680a-0616-11f1-bf55-fa163e6f1404 is removed from agent 10.0.0.240 2026-02-10 00:23:27,587 - INFO - Verified that pod ctest-pod-25423383 is removed in agent 2026-02-10 00:23:27,587 - INFO - Deleting namespace ctest-non-default-44517828 2026-02-10 00:23:27,606 - DEBUG - Namespace ctest-non-default-44517828 still in k8s 2026-02-10 00:23:29,614 - DEBUG - Namespace ctest-non-default-44517828 still in k8s 2026-02-10 00:23:31,619 - DEBUG - Namespace ctest-non-default-44517828 still in k8s 2026-02-10 00:23:33,623 - DEBUG - Namespace ctest-non-default-44517828 is not in k8s 2026-02-10 00:23:33,623 - DEBUG - No need to check project deletion in contrail-api 2026-02-10 00:23:33,623 - INFO - Deleting namespace ctest-new-default-93282662 2026-02-10 00:23:33,634 - DEBUG - Namespace ctest-new-default-93282662 still in k8s 2026-02-10 00:23:35,638 - DEBUG - Namespace ctest-new-default-93282662 still in k8s 2026-02-10 00:23:37,642 - DEBUG - Namespace ctest-new-default-93282662 still in k8s 2026-02-10 00:23:39,646 - DEBUG - Namespace ctest-new-default-93282662 is not in k8s 2026-02-10 00:23:39,646 - DEBUG - No need to check project deletion in contrail-api