2025-09-15 01:52:27,680 - INFO - Using existing project ['default-domain', 'k8s-default'](5e080702-fe61-4589-9d8a-12bed7c59da5) 2025-09-15 01:52:27,698 - DEBUG - Namespace ctest-new-default-28554716 not present 2025-09-15 01:52:27,712 - INFO - Created namespace ctest-new-default-28554716 2025-09-15 01:52:30,716 - DEBUG - Namespace ctest-non-default-56528775 not present 2025-09-15 01:52:30,723 - INFO - Created namespace ctest-non-default-56528775 2025-09-15 01:52:33,747 - DEBUG - Pod ctest-pod-46728405 not present 2025-09-15 01:52:33,747 - INFO - Creating Pod ctest-pod-46728405 2025-09-15 01:52:33,793 - DEBUG - Pod : ctest-pod-46728405 UUID is a12dd869-27c1-4ebe-a5d1-248ab57ddb14 2025-09-15 01:52:33,797 - DEBUG - Pod ctest-pod-25162277 not present 2025-09-15 01:52:33,797 - INFO - Creating Pod ctest-pod-25162277 2025-09-15 01:52:33,826 - DEBUG - Pod : ctest-pod-25162277 UUID is 05f899aa-0b1a-48e2-acfe-e32136d29fe9 2025-09-15 01:52:33,836 - DEBUG - Pod ctest-pod-86644595 not present 2025-09-15 01:52:33,837 - INFO - Creating Pod ctest-pod-86644595 2025-09-15 01:52:33,870 - DEBUG - Pod : ctest-pod-86644595 UUID is 181c2c1a-0af8-41f5-b4a7-2255c62de0ac 2025-09-15 01:52:33,873 - DEBUG - Pod ctest-pod-17904445 not present 2025-09-15 01:52:33,873 - INFO - Creating Pod ctest-pod-17904445 2025-09-15 01:52:33,894 - DEBUG - Pod : ctest-pod-17904445 UUID is 6e178b5d-837d-4d31-abe8-a3f733706601 2025-09-15 01:52:33,898 - DEBUG - Pod ctest-pod-65648601 not present 2025-09-15 01:52:33,899 - INFO - Creating Pod ctest-pod-65648601 2025-09-15 01:52:33,916 - DEBUG - Pod : ctest-pod-65648601 UUID is d3b0c6a2-1c82-4792-9ad4-367c98414ad3 2025-09-15 01:52:33,918 - DEBUG - Pod ctest-pod-40468431 not present 2025-09-15 01:52:33,918 - INFO - Creating Pod ctest-pod-40468431 2025-09-15 01:52:33,939 - DEBUG - Pod : ctest-pod-40468431 UUID is ae23390b-746d-4fa4-b7dd-34e96a97ae42 2025-09-15 01:52:33,942 - DEBUG - Pod ctest-pod-49698694 not present 2025-09-15 01:52:33,943 - INFO - Creating Pod ctest-pod-49698694 2025-09-15 01:52:33,976 - DEBUG - Pod : ctest-pod-49698694 UUID is c8661a1d-d039-45b0-909d-b601414737e5 2025-09-15 01:52:33,981 - DEBUG - Pod ctest-pod-09039883 not present 2025-09-15 01:52:33,981 - INFO - Creating Pod ctest-pod-09039883 2025-09-15 01:52:34,002 - DEBUG - Pod : ctest-pod-09039883 UUID is ed7c5d33-2291-4be9-a537-ed6fe283cbc5 2025-09-15 01:52:34,242 - INFO - Project default-domain:k8s-ctest-new-default-28554716 for Namespace ctest-new-default-28554716 is seen in contrail-api 2025-09-15 01:52:34,242 - DEBUG - Project uuid in Contrail is 01396224-f4d7-4800-a259-f54b870c788a 2025-09-15 01:52:34,404 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_NamespaceDatabaseList?namespace_uuid=eb8ae3c7-df2e-415e-b734-f17e47d50e57 2025-09-15 01:52:34,409 - INFO - Namespace ctest-new-default-28554716 with uuid eb8ae3c7-df2e-415e-b734-f17e47d50e57 found in kube manager 2025-09-15 01:52:34,409 - INFO - Namespace ctest-new-default-28554716 verification passed 2025-09-15 01:52:34,619 - INFO - Project default-domain:k8s-ctest-non-default-56528775 for Namespace ctest-non-default-56528775 is seen in contrail-api 2025-09-15 01:52:34,619 - DEBUG - Project uuid in Contrail is 8861061d-ddd8-450a-99e8-79ca081d32a6 2025-09-15 01:52:34,619 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_NamespaceDatabaseList?namespace_uuid=18507b78-0a9b-4b95-9e5d-ef0e0036aabe 2025-09-15 01:52:34,623 - INFO - Namespace ctest-non-default-56528775 with uuid 18507b78-0a9b-4b95-9e5d-ef0e0036aabe found in kube manager 2025-09-15 01:52:34,623 - INFO - Namespace ctest-non-default-56528775 verification passed 2025-09-15 01:52:34,631 - DEBUG - Pod ctest-pod-46728405 not in running state.Currently in Pending 2025-09-15 01:52:39,640 - DEBUG - Pod ctest-pod-46728405 not in running state.Currently in Pending 2025-09-15 01:52:44,647 - INFO - Pod ctest-pod-46728405 is in running state.Got IP 10.47.255.251 2025-09-15 01:52:44,701 - DEBUG - Pod ctest-pod-46728405 has vmi a6b23936-91d6-11f0-96a8-fa163ecd4331 2025-09-15 01:52:44,701 - INFO - Verified pod ctest-pod-46728405 in contrail-api 2025-09-15 01:52:44,711 - DEBUG - VMI a6b23936-91d6-11f0-96a8-fa163ecd4331 is active in agent 10.0.0.241 2025-09-15 01:52:44,711 - INFO - Verified Pod ctest-pod-46728405 in agent 10.0.0.241 2025-09-15 01:52:44,711 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=a12dd869-27c1-4ebe-a5d1-248ab57ddb14 2025-09-15 01:52:44,715 - INFO - Pod ctest-pod-46728405 with uuid a12dd869-27c1-4ebe-a5d1-248ab57ddb14 found in kube manager 2025-09-15 01:52:44,715 - INFO - Pod ctest-pod-46728405 verification passed 2025-09-15 01:52:44,721 - INFO - Pod ctest-pod-25162277 is in running state.Got IP 10.47.255.250 2025-09-15 01:52:44,732 - DEBUG - Pod ctest-pod-25162277 has vmi a72b30ca-91d6-11f0-96a8-fa163ecd4331 2025-09-15 01:52:44,733 - INFO - Verified pod ctest-pod-25162277 in contrail-api 2025-09-15 01:52:44,743 - DEBUG - VMI a72b30ca-91d6-11f0-96a8-fa163ecd4331 is active in agent 10.0.0.241 2025-09-15 01:52:44,743 - INFO - Verified Pod ctest-pod-25162277 in agent 10.0.0.241 2025-09-15 01:52:44,743 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=05f899aa-0b1a-48e2-acfe-e32136d29fe9 2025-09-15 01:52:44,746 - INFO - Pod ctest-pod-25162277 with uuid 05f899aa-0b1a-48e2-acfe-e32136d29fe9 found in kube manager 2025-09-15 01:52:44,746 - INFO - Pod ctest-pod-25162277 verification passed 2025-09-15 01:52:44,752 - DEBUG - Pod ctest-pod-86644595 not in running state.Currently in Pending 2025-09-15 01:52:49,759 - INFO - Pod ctest-pod-86644595 is in running state.Got IP 10.47.255.249 2025-09-15 01:52:49,813 - DEBUG - Pod ctest-pod-86644595 has vmi a7bfd90a-91d6-11f0-96a8-fa163ecd4331 2025-09-15 01:52:49,813 - INFO - Verified pod ctest-pod-86644595 in contrail-api 2025-09-15 01:52:49,826 - DEBUG - VMI a7bfd90a-91d6-11f0-96a8-fa163ecd4331 is active in agent 10.0.0.241 2025-09-15 01:52:49,826 - INFO - Verified Pod ctest-pod-86644595 in agent 10.0.0.241 2025-09-15 01:52:49,826 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=181c2c1a-0af8-41f5-b4a7-2255c62de0ac 2025-09-15 01:52:49,830 - INFO - Pod ctest-pod-86644595 with uuid 181c2c1a-0af8-41f5-b4a7-2255c62de0ac found in kube manager 2025-09-15 01:52:49,830 - INFO - Pod ctest-pod-86644595 verification passed 2025-09-15 01:52:49,837 - INFO - Pod ctest-pod-17904445 is in running state.Got IP 10.47.255.248 2025-09-15 01:52:49,848 - DEBUG - Pod ctest-pod-17904445 has vmi a82e441c-91d6-11f0-96a8-fa163ecd4331 2025-09-15 01:52:49,848 - INFO - Verified pod ctest-pod-17904445 in contrail-api 2025-09-15 01:52:49,860 - DEBUG - VMI a82e441c-91d6-11f0-96a8-fa163ecd4331 is active in agent 10.0.0.241 2025-09-15 01:52:49,860 - INFO - Verified Pod ctest-pod-17904445 in agent 10.0.0.241 2025-09-15 01:52:49,860 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=6e178b5d-837d-4d31-abe8-a3f733706601 2025-09-15 01:52:49,864 - INFO - Pod ctest-pod-17904445 with uuid 6e178b5d-837d-4d31-abe8-a3f733706601 found in kube manager 2025-09-15 01:52:49,864 - INFO - Pod ctest-pod-17904445 verification passed 2025-09-15 01:52:49,870 - INFO - Pod ctest-pod-65648601 is in running state.Got IP 10.47.255.247 2025-09-15 01:52:49,924 - DEBUG - Pod ctest-pod-65648601 has vmi a8aedc62-91d6-11f0-96a8-fa163ecd4331 2025-09-15 01:52:49,924 - INFO - Verified pod ctest-pod-65648601 in contrail-api 2025-09-15 01:52:49,937 - DEBUG - VMI a8aedc62-91d6-11f0-96a8-fa163ecd4331 is active in agent 10.0.0.241 2025-09-15 01:52:49,937 - INFO - Verified Pod ctest-pod-65648601 in agent 10.0.0.241 2025-09-15 01:52:49,937 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=d3b0c6a2-1c82-4792-9ad4-367c98414ad3 2025-09-15 01:52:49,942 - INFO - Pod ctest-pod-65648601 with uuid d3b0c6a2-1c82-4792-9ad4-367c98414ad3 found in kube manager 2025-09-15 01:52:49,942 - INFO - Pod ctest-pod-65648601 verification passed 2025-09-15 01:52:49,948 - DEBUG - Pod ctest-pod-49698694 not in running state.Currently in Pending 2025-09-15 01:52:54,956 - INFO - Pod ctest-pod-49698694 is in running state.Got IP 10.47.255.245 2025-09-15 01:52:54,968 - DEBUG - Pod ctest-pod-49698694 has vmi a98258f8-91d6-11f0-96a8-fa163ecd4331 2025-09-15 01:52:54,968 - INFO - Verified pod ctest-pod-49698694 in contrail-api 2025-09-15 01:52:54,984 - DEBUG - VMI a98258f8-91d6-11f0-96a8-fa163ecd4331 is active in agent 10.0.0.241 2025-09-15 01:52:54,985 - INFO - Verified Pod ctest-pod-49698694 in agent 10.0.0.241 2025-09-15 01:52:54,985 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=c8661a1d-d039-45b0-909d-b601414737e5 2025-09-15 01:52:54,989 - INFO - Pod ctest-pod-49698694 with uuid c8661a1d-d039-45b0-909d-b601414737e5 found in kube manager 2025-09-15 01:52:54,989 - INFO - Pod ctest-pod-49698694 verification passed 2025-09-15 01:52:54,996 - INFO - Pod ctest-pod-40468431 is in running state.Got IP 10.47.255.246 2025-09-15 01:52:55,048 - DEBUG - Pod ctest-pod-40468431 has vmi a9157706-91d6-11f0-96a8-fa163ecd4331 2025-09-15 01:52:55,048 - INFO - Verified pod ctest-pod-40468431 in contrail-api 2025-09-15 01:52:55,060 - DEBUG - VMI a9157706-91d6-11f0-96a8-fa163ecd4331 is active in agent 10.0.0.241 2025-09-15 01:52:55,061 - INFO - Verified Pod ctest-pod-40468431 in agent 10.0.0.241 2025-09-15 01:52:55,061 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=ae23390b-746d-4fa4-b7dd-34e96a97ae42 2025-09-15 01:52:55,065 - INFO - Pod ctest-pod-40468431 with uuid ae23390b-746d-4fa4-b7dd-34e96a97ae42 found in kube manager 2025-09-15 01:52:55,065 - INFO - Pod ctest-pod-40468431 verification passed 2025-09-15 01:52:55,071 - INFO - Pod ctest-pod-09039883 is in running state.Got IP 10.47.255.244 2025-09-15 01:52:55,082 - DEBUG - Pod ctest-pod-09039883 has vmi a9e8f8d8-91d6-11f0-96a8-fa163ecd4331 2025-09-15 01:52:55,082 - INFO - Verified pod ctest-pod-09039883 in contrail-api 2025-09-15 01:52:55,092 - DEBUG - VMI a9e8f8d8-91d6-11f0-96a8-fa163ecd4331 is active in agent 10.0.0.241 2025-09-15 01:52:55,092 - INFO - Verified Pod ctest-pod-09039883 in agent 10.0.0.241 2025-09-15 01:52:55,093 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_VirtualMachineDatabaseList?x=ed7c5d33-2291-4be9-a537-ed6fe283cbc5 2025-09-15 01:52:55,096 - INFO - Pod ctest-pod-09039883 with uuid ed7c5d33-2291-4be9-a537-ed6fe283cbc5 found in kube manager 2025-09-15 01:52:55,096 - INFO - Pod ctest-pod-09039883 verification passed 2025-09-15 01:52:55,096 - INFO - ================================================================================ 2025-09-15 01:52:55,096 - INFO - STARTING TEST : test_ingress_rule_on_pod_with_service 2025-09-15 01:52:55,096 - INFO - TEST DESCRIPTION : Verify that ingress rules on a pod work fine when pods are accessed via a service Steps: 1. Create a service in namespace 1 over a pod which are endpoints of a service 2. Create a network policy to allow 1 pod of namespace 1 as ingress. 3. Verify that service respond correctly to the specified pod in the policy and service do not respond to anyone else. 4. Extend the policy and add the namespaceselector as namespace 2. 5. Verify that service become accessible to the pods of namespace 2. 2025-09-15 01:52:55,303 - DEBUG - Nothing to compare xmpp stats {'10.0.0.241': {'10.0.0.33': '0'}} with 2025-09-15 01:52:55,303 - INFO - Initial checks done. Running the testcase now 2025-09-15 01:52:55,303 - INFO - 2025-09-15 01:52:55,366 - DEBUG - [Pod ctest-pod-46728405] Cmd: echo ctest-pod-46728405 > /usr/share/nginx/html/index.html, Output: 2025-09-15 01:52:55,403 - DEBUG - [Pod ctest-pod-25162277] Cmd: echo ctest-pod-25162277 > /usr/share/nginx/html/index.html, Output: 2025-09-15 01:52:55,453 - DEBUG - [Pod ctest-pod-86644595] Cmd: echo ctest-pod-86644595 > /usr/share/nginx/html/index.html, Output: 2025-09-15 01:52:55,519 - DEBUG - [Pod ctest-pod-17904445] Cmd: echo ctest-pod-17904445 > /usr/share/nginx/html/index.html, Output: 2025-09-15 01:52:55,524 - DEBUG - Service ctest-nginx-svc-70191969 not present 2025-09-15 01:52:55,524 - INFO - Creating service ctest-nginx-svc-70191969 2025-09-15 01:52:55,535 - INFO - Created Service ctest-nginx-svc-70191969 2025-09-15 01:52:55,543 - WARNING - Service UUID 59c4f95c-d876-4f31-9076-719c75a08be9 not yet found in contrail-api 2025-09-15 01:52:56,550 - INFO - Validated that Service ctest-nginx-svc-70191969 is seen in contrail-api 2025-09-15 01:52:56,551 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_LoadbalancerDatabaseList?x=59c4f95c-d876-4f31-9076-719c75a08be9 2025-09-15 01:52:56,556 - INFO - Service ctest-nginx-svc-70191969 with uuid 59c4f95c-d876-4f31-9076-719c75a08be9 found in kube manager 2025-09-15 01:52:56,556 - INFO - Service ctest-nginx-svc-70191969 verification passed 2025-09-15 01:52:57,644 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-25162277 2025-09-15 01:52:57,644 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:52:57,644 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-25162277 2025-09-15 01:52:57,696 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:52:57,696 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:52:57,696 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:52:57,751 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:52:57,751 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:52:57,751 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:52:57,751 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-46728405': 1, 'ctest-pod-25162277': 2} 2025-09-15 01:52:57,808 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:52:57,808 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:52:57,808 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:52:57,861 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:52:57,862 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:52:57,862 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:52:57,862 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-46728405': 1, 'ctest-pod-25162277': 1} 2025-09-15 01:52:57,866 - DEBUG - Network policy ingress-on-service not present 2025-09-15 01:52:57,866 - INFO - Creating Network Policy ingress-on-service 2025-09-15 01:52:57,878 - INFO - Network policy found in k8s 2025-09-15 01:52:57,878 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=ff3f9436-d944-447b-a61b-5fe8c34556d8 2025-09-15 01:52:57,883 - INFO - Network Policy ingress-on-service with uuid ff3f9436-d944-447b-a61b-5fe8c34556d8 found in kube manager 2025-09-15 01:52:57,911 - INFO - Network policy with name default-policy-management:k8s-allowall found in agent 2025-09-15 01:52:57,911 - INFO - Network policy with name default-policy-management:k8s-allowall associated with default ks8 2025-09-15 01:52:57,911 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=ff3f9436-d944-447b-a61b-5fe8c34556d8 2025-09-15 01:52:57,931 - WARNING - Network policy with name ingress-on-service not found in agent 2025-09-15 01:52:59,932 - DEBUG - Requesting: http://10.0.0.33:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=ff3f9436-d944-447b-a61b-5fe8c34556d8 2025-09-15 01:52:59,956 - INFO - Network policy with name ingress-on-service found in agent 2025-09-15 01:52:59,957 - INFO - Network Policy ingress-on-service verification passed 2025-09-15 01:53:05,004 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:05,004 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:05,004 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:10,051 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:10,051 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:10,051 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:15,103 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:15,103 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:15,104 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:20,156 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:20,156 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:20,156 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:25,203 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:25,203 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:25,203 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:30,256 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:30,256 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:30,256 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:35,310 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:35,310 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:35,310 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:40,363 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:40,363 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:40,363 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:45,415 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:45,473 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:45,473 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:50,527 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:50,527 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:50,527 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:50,527 - INFO - As expected, responses not seen from podsHits : {'ctest-pod-46728405': 0, 'ctest-pod-25162277': 0} 2025-09-15 01:53:50,592 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:53:50,592 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:53:50,592 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:53:50,646 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:53:50,646 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:53:50,646 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:53:50,704 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:53:50,704 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:53:50,705 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:53:50,763 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:53:50,763 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:53:50,763 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:53:50,814 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-25162277 2025-09-15 01:53:50,814 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:53:50,814 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-25162277 2025-09-15 01:53:50,865 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) ctest-pod-46728405 writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:53:50,865 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:53:50,865 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) ctest-pod-46728405 writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:53:50,865 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-46728405': 1, 'ctest-pod-25162277': 5} 2025-09-15 01:53:55,912 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:55,912 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:53:55,912 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:00,966 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:00,966 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:00,966 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:06,018 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:06,018 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:06,019 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:11,067 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:11,067 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:11,067 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:11,486 - WARNING - ApiException is caught (0) Reason: Connection is already closed. . Retrying... 2025-09-15 01:54:21,929 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:21,929 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:21,929 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:26,980 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:26,980 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:26,980 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:32,035 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:32,035 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:32,035 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:37,095 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:37,095 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:37,095 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:42,143 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:42,143 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:42,143 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:47,196 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:47,196 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:47,196 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:54:47,197 - INFO - As expected, responses not seen from podsHits : {'ctest-pod-46728405': 0, 'ctest-pod-25162277': 0} 2025-09-15 01:54:49,244 - DEBUG - [Pod ctest-pod-40468431] Cmd: ping -c 3 10.47.255.247, Output: PING 10.47.255.247 (10.47.255.247): 56 data bytes 64 bytes from 10.47.255.247: seq=0 ttl=63 time=0.762 ms 64 bytes from 10.47.255.247: seq=1 ttl=63 time=0.073 ms 64 bytes from 10.47.255.247: seq=2 ttl=63 time=0.089 ms --- 10.47.255.247 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.073/0.308/0.762 ms 2025-09-15 01:54:49,244 - INFO - Ping check to IP 10.47.255.247 from pod ctest-pod-40468431 with expectation True passed 2025-09-15 01:54:51,300 - DEBUG - [Pod ctest-pod-65648601] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes 64 bytes from 10.47.255.246: seq=0 ttl=63 time=1.557 ms 64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.076 ms 64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.075 ms --- 10.47.255.246 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.075/0.569/1.557 ms 2025-09-15 01:54:51,300 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-65648601 with expectation True passed 2025-09-15 01:54:53,347 - DEBUG - [Pod ctest-pod-49698694] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes 64 bytes from 10.47.255.246: seq=0 ttl=63 time=0.382 ms 64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.064 ms 64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.095 ms --- 10.47.255.246 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.064/0.180/0.382 ms 2025-09-15 01:54:53,347 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-49698694 with expectation True passed 2025-09-15 01:54:53,413 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.47.255.251 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.251 (10.47.255.251:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2025-09-15 01:54:53,413 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.47.255.251 -O /dev/null -T 5 -t 1 passed 2025-09-15 01:54:53,413 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: Connecting to 10.47.255.251 (10.47.255.251:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2025-09-15 01:54:53,413 - INFO - wget check of of http://10.47.255.251 from ctest-pod-40468431 passed 2025-09-15 01:54:53,465 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2025-09-15 01:54:53,466 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2025-09-15 01:54:53,466 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2025-09-15 01:54:53,466 - INFO - wget check of of http://10.47.255.250 from ctest-pod-40468431 passed 2025-09-15 01:54:58,523 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) wget: download timed out 2025-09-15 01:54:58,523 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 failed. Output :Connecting to 10.47.255.250 (10.47.255.250:80) wget: download timed out 2025-09-15 01:54:58,523 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) wget: download timed out 2025-09-15 01:54:58,523 - INFO - wget check of of http://10.47.255.250 from ctest-pod-65648601 passed 2025-09-15 01:55:03,582 - DEBUG - [Pod ctest-pod-09039883] Cmd: wget http://10.47.255.251 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.251 (10.47.255.251:80) wget: download timed out 2025-09-15 01:55:03,582 - DEBUG - [Pod Pod ctest-pod-09039883] Cmd wget http://10.47.255.251 -O /dev/null -T 5 -t 1 failed. Output :Connecting to 10.47.255.251 (10.47.255.251:80) wget: download timed out 2025-09-15 01:55:03,582 - DEBUG - [Pod Pod ctest-pod-09039883] Cmd output: Connecting to 10.47.255.251 (10.47.255.251:80) wget: download timed out 2025-09-15 01:55:03,582 - INFO - wget check of of http://10.47.255.251 from ctest-pod-09039883 passed 2025-09-15 01:55:03,591 - INFO - Updating Network Policy ingress-on-service 2025-09-15 01:55:08,643 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:08,643 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:08,643 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:13,687 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:13,687 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:13,687 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:18,745 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:18,745 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:18,745 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:23,794 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:23,795 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:23,795 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:28,852 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:28,852 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:28,852 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:33,905 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:33,905 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:33,905 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:38,972 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:38,972 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:38,972 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:44,023 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:44,023 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:44,023 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:49,076 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:49,077 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:49,077 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:54,126 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:54,126 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:54,126 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) wget: download timed out 2025-09-15 01:55:54,126 - INFO - As expected, responses not seen from podsHits : {'ctest-pod-46728405': 0, 'ctest-pod-25162277': 0} 2025-09-15 01:55:54,176 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:55:54,177 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:55:54,177 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:55:54,224 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:55:54,224 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:55:54,224 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: ctest-pod-25162277 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:55:54,224 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-46728405': 1, 'ctest-pod-25162277': 1} 2025-09-15 01:55:54,280 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:55:54,280 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:55:54,280 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:55:54,336 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:55:54,337 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:55:54,337 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:55:54,390 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:55:54,390 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:55:54,390 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: ctest-pod-46728405 Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2025-09-15 01:55:54,436 - DEBUG - [Pod ctest-pod-49698694] Cmd: wget http://10.106.193.131:80/ -O - -T 5 -t 1 , Output: Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-25162277 2025-09-15 01:55:54,437 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd wget http://10.106.193.131:80/ -O - -T 5 -t 1 passed 2025-09-15 01:55:54,437 - DEBUG - [Pod Pod ctest-pod-49698694] Cmd output: Connecting to 10.106.193.131:80 (10.106.193.131:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-25162277 2025-09-15 01:55:54,437 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-46728405': 3, 'ctest-pod-25162277': 1} 2025-09-15 01:55:56,497 - DEBUG - [Pod ctest-pod-40468431] Cmd: ping -c 3 10.47.255.247, Output: PING 10.47.255.247 (10.47.255.247): 56 data bytes 64 bytes from 10.47.255.247: seq=0 ttl=63 time=1.302 ms 64 bytes from 10.47.255.247: seq=1 ttl=63 time=0.061 ms 64 bytes from 10.47.255.247: seq=2 ttl=63 time=0.082 ms --- 10.47.255.247 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.061/0.481/1.302 ms 2025-09-15 01:55:56,498 - INFO - Ping check to IP 10.47.255.247 from pod ctest-pod-40468431 with expectation True passed 2025-09-15 01:55:58,552 - DEBUG - [Pod ctest-pod-65648601] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes 64 bytes from 10.47.255.246: seq=0 ttl=63 time=0.524 ms 64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.078 ms 64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.089 ms --- 10.47.255.246 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.078/0.230/0.524 ms 2025-09-15 01:55:58,552 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-65648601 with expectation True passed 2025-09-15 01:56:00,603 - DEBUG - [Pod ctest-pod-49698694] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes 64 bytes from 10.47.255.246: seq=0 ttl=63 time=0.971 ms 64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.074 ms 64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.098 ms --- 10.47.255.246 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.074/0.381/0.971 ms 2025-09-15 01:56:00,603 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-49698694 with expectation True passed 2025-09-15 01:56:00,656 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.47.255.251 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.251 (10.47.255.251:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2025-09-15 01:56:00,656 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.47.255.251 -O /dev/null -T 5 -t 1 passed 2025-09-15 01:56:00,656 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: Connecting to 10.47.255.251 (10.47.255.251:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2025-09-15 01:56:00,656 - INFO - wget check of of http://10.47.255.251 from ctest-pod-40468431 passed 2025-09-15 01:56:00,705 - DEBUG - [Pod ctest-pod-40468431] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2025-09-15 01:56:00,705 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2025-09-15 01:56:00,705 - DEBUG - [Pod Pod ctest-pod-40468431] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2025-09-15 01:56:00,705 - INFO - wget check of of http://10.47.255.250 from ctest-pod-40468431 passed 2025-09-15 01:56:05,757 - DEBUG - [Pod ctest-pod-65648601] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) wget: download timed out 2025-09-15 01:56:05,757 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 failed. Output :Connecting to 10.47.255.250 (10.47.255.250:80) wget: download timed out 2025-09-15 01:56:05,757 - DEBUG - [Pod Pod ctest-pod-65648601] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) wget: download timed out 2025-09-15 01:56:05,757 - INFO - wget check of of http://10.47.255.250 from ctest-pod-65648601 passed 2025-09-15 01:56:05,812 - DEBUG - [Pod ctest-pod-09039883] Cmd: wget http://10.47.255.251 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.251 (10.47.255.251:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2025-09-15 01:56:05,812 - DEBUG - [Pod Pod ctest-pod-09039883] Cmd wget http://10.47.255.251 -O /dev/null -T 5 -t 1 passed 2025-09-15 01:56:05,812 - DEBUG - [Pod Pod ctest-pod-09039883] Cmd output: Connecting to 10.47.255.251 (10.47.255.251:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2025-09-15 01:56:05,812 - INFO - wget check of of http://10.47.255.251 from ctest-pod-09039883 passed 2025-09-15 01:56:05,812 - INFO - Deleting Network Policy : ingress-on-service 2025-09-15 01:56:05,813 - INFO - Deleting service : ctest-nginx-svc-70191969 2025-09-15 01:56:06,053 - DEBUG - No XMPP flaps were noticed during the test