2025-01-02 20:57:21,149 - INFO - Using existing project ['default-domain', 'k8s-default'](a148ec8c-a5ca-46d4-9987-03d6ff11dd67)
2025-01-02 20:57:21,194 - DEBUG - Namespace ctest-new-default-76592153 not present
2025-01-02 20:57:21,224 - INFO - Created namespace ctest-new-default-76592153
2025-01-02 20:57:24,229 - DEBUG - Namespace ctest-non-default-30747418 not present
2025-01-02 20:57:24,274 - INFO - Created namespace ctest-non-default-30747418
2025-01-02 20:57:27,290 - DEBUG - Pod ctest-pod-16149351 not present
2025-01-02 20:57:27,290 - INFO - Creating Pod ctest-pod-16149351
2025-01-02 20:57:27,352 - DEBUG - Pod : ctest-pod-16149351 UUID is 095bf4aa-3875-4eae-8cfc-948692641635
2025-01-02 20:57:27,355 - DEBUG - Pod ctest-pod-01498961 not present
2025-01-02 20:57:27,355 - INFO - Creating Pod ctest-pod-01498961
2025-01-02 20:57:27,413 - DEBUG - Pod : ctest-pod-01498961 UUID is 1f6b0126-165e-461b-83f1-456134a25450
2025-01-02 20:57:27,418 - DEBUG - Pod ctest-pod-13703445 not present
2025-01-02 20:57:27,419 - INFO - Creating Pod ctest-pod-13703445
2025-01-02 20:57:27,460 - DEBUG - Pod : ctest-pod-13703445 UUID is 354b8260-2192-41f4-b049-dd41a21f6c37
2025-01-02 20:57:27,468 - DEBUG - Pod ctest-pod-90850542 not present
2025-01-02 20:57:27,468 - INFO - Creating Pod ctest-pod-90850542
2025-01-02 20:57:27,493 - DEBUG - Pod : ctest-pod-90850542 UUID is 54943c50-cb6c-4c21-b6d9-9f36cc4980bb
2025-01-02 20:57:27,496 - DEBUG - Pod ctest-pod-76539307 not present
2025-01-02 20:57:27,496 - INFO - Creating Pod ctest-pod-76539307
2025-01-02 20:57:27,526 - DEBUG - Pod : ctest-pod-76539307 UUID is 23f74e67-cc80-4c34-b25d-26844b4d339f
2025-01-02 20:57:27,529 - DEBUG - Pod ctest-pod-09099449 not present
2025-01-02 20:57:27,529 - INFO - Creating Pod ctest-pod-09099449
2025-01-02 20:57:27,585 - DEBUG - Pod : ctest-pod-09099449 UUID is cf80041c-44ba-498f-bdc5-4dbacd3cf5e0
2025-01-02 20:57:27,596 - DEBUG - Pod ctest-pod-54167827 not present
2025-01-02 20:57:27,596 - INFO - Creating Pod ctest-pod-54167827
2025-01-02 20:57:27,626 - DEBUG - Pod : ctest-pod-54167827 UUID is f61165a0-d0e4-4919-bc5d-40ab352ff1da
2025-01-02 20:57:27,635 - DEBUG - Pod ctest-pod-28354863 not present
2025-01-02 20:57:27,635 - INFO - Creating Pod ctest-pod-28354863
2025-01-02 20:57:27,657 - DEBUG - Pod : ctest-pod-28354863 UUID is ee3d708c-6873-40c1-89af-b552b0bb2457
2025-01-02 20:57:27,900 - INFO - Project default-domain:k8s-ctest-new-default-76592153 for Namespace ctest-new-default-76592153 is seen in contrail-api
2025-01-02 20:57:27,900 - DEBUG - Project uuid in Contrail is 89df9c30-7a8b-4296-a7cf-c57194814c59
2025-01-02 20:57:28,031 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_NamespaceDatabaseList?namespace_uuid=cee9de7b-5902-4e2a-90b8-880cc9dcf7af
2025-01-02 20:57:28,051 - INFO - Namespace ctest-new-default-76592153 with uuid cee9de7b-5902-4e2a-90b8-880cc9dcf7af found in kube manager
2025-01-02 20:57:28,052 - INFO - Namespace ctest-new-default-76592153 verification passed
2025-01-02 20:57:28,208 - INFO - Project default-domain:k8s-ctest-non-default-30747418 for Namespace ctest-non-default-30747418 is seen in contrail-api
2025-01-02 20:57:28,208 - DEBUG - Project uuid in Contrail is 1908b961-876c-48c0-8640-2bac74bfc438
2025-01-02 20:57:28,208 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_NamespaceDatabaseList?namespace_uuid=dd093bfb-027f-46cc-8ad0-1796b124a0fe
2025-01-02 20:57:28,221 - INFO - Namespace ctest-non-default-30747418 with uuid dd093bfb-027f-46cc-8ad0-1796b124a0fe found in kube manager
2025-01-02 20:57:28,221 - INFO - Namespace ctest-non-default-30747418 verification passed
2025-01-02 20:57:28,229 - DEBUG - Pod ctest-pod-16149351 not in running state.Currently in Pending
2025-01-02 20:57:33,234 - DEBUG - Pod ctest-pod-16149351 not in running state.Currently in Pending
2025-01-02 20:57:38,240 - DEBUG - Pod ctest-pod-16149351 not in running state.Currently in Pending
2025-01-02 20:57:43,246 - INFO - Pod ctest-pod-16149351 is in running state.Got IP 10.47.255.251
2025-01-02 20:57:43,262 - DEBUG - Pod ctest-pod-16149351 has vmi 2d9e6720-c94c-11ef-9f85-fa163e8eb557
2025-01-02 20:57:43,263 - INFO - Verified pod ctest-pod-16149351 in contrail-api
2025-01-02 20:57:43,279 - DEBUG - VMI 2d9e6720-c94c-11ef-9f85-fa163e8eb557 is active in agent 10.0.0.37
2025-01-02 20:57:43,279 - INFO - Verified Pod ctest-pod-16149351 in agent 10.0.0.37
2025-01-02 20:57:43,279 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_VirtualMachineDatabaseList?x=095bf4aa-3875-4eae-8cfc-948692641635
2025-01-02 20:57:43,285 - INFO - Pod ctest-pod-16149351 with uuid 095bf4aa-3875-4eae-8cfc-948692641635 found in kube manager
2025-01-02 20:57:43,286 - INFO - Pod ctest-pod-16149351 verification passed
2025-01-02 20:57:43,292 - INFO - Pod ctest-pod-01498961 is in running state.Got IP 10.47.255.250
2025-01-02 20:57:43,352 - DEBUG - Pod ctest-pod-01498961 has vmi 2e20f474-c94c-11ef-9f85-fa163e8eb557
2025-01-02 20:57:43,352 - INFO - Verified pod ctest-pod-01498961 in contrail-api
2025-01-02 20:57:43,368 - DEBUG - VMI 2e20f474-c94c-11ef-9f85-fa163e8eb557 is active in agent 10.0.0.37
2025-01-02 20:57:43,368 - INFO - Verified Pod ctest-pod-01498961 in agent 10.0.0.37
2025-01-02 20:57:43,369 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_VirtualMachineDatabaseList?x=1f6b0126-165e-461b-83f1-456134a25450
2025-01-02 20:57:43,373 - INFO - Pod ctest-pod-01498961 with uuid 1f6b0126-165e-461b-83f1-456134a25450 found in kube manager
2025-01-02 20:57:43,374 - INFO - Pod ctest-pod-01498961 verification passed
2025-01-02 20:57:43,381 - INFO - Pod ctest-pod-13703445 is in running state.Got IP 10.47.255.249
2025-01-02 20:57:43,437 - DEBUG - Pod ctest-pod-13703445 has vmi 2ed2d086-c94c-11ef-9f85-fa163e8eb557
2025-01-02 20:57:43,437 - INFO - Verified pod ctest-pod-13703445 in contrail-api
2025-01-02 20:57:43,454 - DEBUG - VMI 2ed2d086-c94c-11ef-9f85-fa163e8eb557 is active in agent 10.0.0.37
2025-01-02 20:57:43,455 - INFO - Verified Pod ctest-pod-13703445 in agent 10.0.0.37
2025-01-02 20:57:43,455 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_VirtualMachineDatabaseList?x=354b8260-2192-41f4-b049-dd41a21f6c37
2025-01-02 20:57:43,460 - INFO - Pod ctest-pod-13703445 with uuid 354b8260-2192-41f4-b049-dd41a21f6c37 found in kube manager
2025-01-02 20:57:43,460 - INFO - Pod ctest-pod-13703445 verification passed
2025-01-02 20:57:43,466 - INFO - Pod ctest-pod-90850542 is in running state.Got IP 10.47.255.248
2025-01-02 20:57:43,480 - DEBUG - Pod ctest-pod-90850542 has vmi 2f5400ca-c94c-11ef-9f85-fa163e8eb557
2025-01-02 20:57:43,480 - INFO - Verified pod ctest-pod-90850542 in contrail-api
2025-01-02 20:57:43,493 - DEBUG - VMI 2f5400ca-c94c-11ef-9f85-fa163e8eb557 is active in agent 10.0.0.37
2025-01-02 20:57:43,494 - INFO - Verified Pod ctest-pod-90850542 in agent 10.0.0.37
2025-01-02 20:57:43,494 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_VirtualMachineDatabaseList?x=54943c50-cb6c-4c21-b6d9-9f36cc4980bb
2025-01-02 20:57:43,499 - INFO - Pod ctest-pod-90850542 with uuid 54943c50-cb6c-4c21-b6d9-9f36cc4980bb found in kube manager
2025-01-02 20:57:43,499 - INFO - Pod ctest-pod-90850542 verification passed
2025-01-02 20:57:43,506 - DEBUG - Pod ctest-pod-76539307 not in running state.Currently in Pending
2025-01-02 20:57:48,513 - INFO - Pod ctest-pod-76539307 is in running state.Got IP 10.47.255.247
2025-01-02 20:57:48,569 - DEBUG - Pod ctest-pod-76539307 has vmi 2fe85c20-c94c-11ef-9f85-fa163e8eb557
2025-01-02 20:57:48,570 - INFO - Verified pod ctest-pod-76539307 in contrail-api
2025-01-02 20:57:48,583 - DEBUG - VMI 2fe85c20-c94c-11ef-9f85-fa163e8eb557 is active in agent 10.0.0.37
2025-01-02 20:57:48,583 - INFO - Verified Pod ctest-pod-76539307 in agent 10.0.0.37
2025-01-02 20:57:48,583 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_VirtualMachineDatabaseList?x=23f74e67-cc80-4c34-b25d-26844b4d339f
2025-01-02 20:57:48,588 - INFO - Pod ctest-pod-76539307 with uuid 23f74e67-cc80-4c34-b25d-26844b4d339f found in kube manager
2025-01-02 20:57:48,588 - INFO - Pod ctest-pod-76539307 verification passed
2025-01-02 20:57:48,593 - INFO - Pod ctest-pod-54167827 is in running state.Got IP 10.47.255.245
2025-01-02 20:57:48,606 - DEBUG - Pod ctest-pod-54167827 has vmi 30d41dcc-c94c-11ef-9f85-fa163e8eb557
2025-01-02 20:57:48,607 - INFO - Verified pod ctest-pod-54167827 in contrail-api
2025-01-02 20:57:48,619 - DEBUG - VMI 30d41dcc-c94c-11ef-9f85-fa163e8eb557 is active in agent 10.0.0.37
2025-01-02 20:57:48,619 - INFO - Verified Pod ctest-pod-54167827 in agent 10.0.0.37
2025-01-02 20:57:48,620 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_VirtualMachineDatabaseList?x=f61165a0-d0e4-4919-bc5d-40ab352ff1da
2025-01-02 20:57:48,624 - INFO - Pod ctest-pod-54167827 with uuid f61165a0-d0e4-4919-bc5d-40ab352ff1da found in kube manager
2025-01-02 20:57:48,624 - INFO - Pod ctest-pod-54167827 verification passed
2025-01-02 20:57:48,630 - INFO - Pod ctest-pod-09099449 is in running state.Got IP 10.47.255.246
2025-01-02 20:57:48,685 - DEBUG - Pod ctest-pod-09099449 has vmi 305a3890-c94c-11ef-9f85-fa163e8eb557
2025-01-02 20:57:48,685 - INFO - Verified pod ctest-pod-09099449 in contrail-api
2025-01-02 20:57:48,698 - DEBUG - VMI 305a3890-c94c-11ef-9f85-fa163e8eb557 is active in agent 10.0.0.37
2025-01-02 20:57:48,698 - INFO - Verified Pod ctest-pod-09099449 in agent 10.0.0.37
2025-01-02 20:57:48,699 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_VirtualMachineDatabaseList?x=cf80041c-44ba-498f-bdc5-4dbacd3cf5e0
2025-01-02 20:57:48,703 - INFO - Pod ctest-pod-09099449 with uuid cf80041c-44ba-498f-bdc5-4dbacd3cf5e0 found in kube manager
2025-01-02 20:57:48,703 - INFO - Pod ctest-pod-09099449 verification passed
2025-01-02 20:57:48,716 - INFO - Pod ctest-pod-28354863 is in running state.Got IP 10.47.255.244
2025-01-02 20:57:48,773 - DEBUG - Pod ctest-pod-28354863 has vmi 314b9b4a-c94c-11ef-9f85-fa163e8eb557
2025-01-02 20:57:48,773 - INFO - Verified pod ctest-pod-28354863 in contrail-api
2025-01-02 20:57:48,787 - DEBUG - VMI 314b9b4a-c94c-11ef-9f85-fa163e8eb557 is active in agent 10.0.0.37
2025-01-02 20:57:48,787 - INFO - Verified Pod ctest-pod-28354863 in agent 10.0.0.37
2025-01-02 20:57:48,787 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_VirtualMachineDatabaseList?x=ee3d708c-6873-40c1-89af-b552b0bb2457
2025-01-02 20:57:48,792 - INFO - Pod ctest-pod-28354863 with uuid ee3d708c-6873-40c1-89af-b552b0bb2457 found in kube manager
2025-01-02 20:57:48,792 - INFO - Pod ctest-pod-28354863 verification passed
2025-01-02 20:57:48,792 - INFO - ================================================================================
2025-01-02 20:57:48,792 - INFO - STARTING TEST : test_ingress_rule_on_pod_with_service
2025-01-02 20:57:48,792 - INFO - TEST DESCRIPTION :
Verify that ingress rules on a pod work fine when pods are accessed via a service
Steps:
1. Create a service in namespace 1 over a pod which are endpoints of a service
2. Create a network policy to allow 1 pod of namespace 1 as ingress.
3. Verify that service respond correctly to the specified pod in the policy and
service do not respond to anyone else.
4. Extend the policy and add the namespaceselector as namespace 2.
5. Verify that service become accessible to the pods of namespace 2.
2025-01-02 20:57:49,019 - DEBUG - Nothing to compare xmpp stats {'10.0.0.37': {'10.0.0.30': '0'}} with
2025-01-02 20:57:49,019 - INFO - Initial checks done. Running the testcase now
2025-01-02 20:57:49,019 - INFO -
2025-01-02 20:57:49,092 - DEBUG - [Pod ctest-pod-16149351] Cmd: echo ctest-pod-16149351 > /usr/share/nginx/html/index.html, Output:
2025-01-02 20:57:49,145 - DEBUG - [Pod ctest-pod-01498961] Cmd: echo ctest-pod-01498961 > /usr/share/nginx/html/index.html, Output:
2025-01-02 20:57:49,201 - DEBUG - [Pod ctest-pod-13703445] Cmd: echo ctest-pod-13703445 > /usr/share/nginx/html/index.html, Output:
2025-01-02 20:57:49,257 - DEBUG - [Pod ctest-pod-90850542] Cmd: echo ctest-pod-90850542 > /usr/share/nginx/html/index.html, Output:
2025-01-02 20:57:49,261 - DEBUG - Service ctest-nginx-svc-49506625 not present
2025-01-02 20:57:49,261 - INFO - Creating service ctest-nginx-svc-49506625
2025-01-02 20:57:49,287 - INFO - Created Service ctest-nginx-svc-49506625
2025-01-02 20:57:49,297 - WARNING - Service UUID e48e05b8-e28e-4848-ba04-9dddbc3d2b3e not yet found in contrail-api
2025-01-02 20:57:50,306 - INFO - Validated that Service ctest-nginx-svc-49506625 is seen in contrail-api
2025-01-02 20:57:50,307 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_LoadbalancerDatabaseList?x=e48e05b8-e28e-4848-ba04-9dddbc3d2b3e
2025-01-02 20:57:50,312 - INFO - Service ctest-nginx-svc-49506625 with uuid e48e05b8-e28e-4848-ba04-9dddbc3d2b3e found in kube manager
2025-01-02 20:57:50,312 - INFO - Service ctest-nginx-svc-49506625 verification passed
2025-01-02 20:57:51,395 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-01498961
2025-01-02 20:57:51,395 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 20:57:51,395 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-01498961
2025-01-02 20:57:51,468 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-01498961
2025-01-02 20:57:51,468 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 20:57:51,468 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-01498961
2025-01-02 20:57:51,525 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 20:57:51,525 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 20:57:51,525 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 20:57:51,573 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-16149351
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 20:57:51,573 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 20:57:51,573 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: ctest-pod-16149351
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 20:57:51,573 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16149351': 1, 'ctest-pod-01498961': 3}
2025-01-02 20:57:51,628 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-01498961
2025-01-02 20:57:51,628 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 20:57:51,628 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-01498961
2025-01-02 20:57:51,688 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-16149351
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 20:57:51,688 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 20:57:51,688 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: ctest-pod-16149351
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 20:57:51,688 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16149351': 1, 'ctest-pod-01498961': 1}
2025-01-02 20:57:51,691 - DEBUG - Network policy ingress-on-service not present
2025-01-02 20:57:51,691 - INFO - Creating Network Policy ingress-on-service
2025-01-02 20:57:51,704 - INFO - Network policy found in k8s
2025-01-02 20:57:51,704 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=aed67be5-d620-4f0c-83b2-f0154c001204
2025-01-02 20:57:51,713 - INFO - Network Policy ingress-on-service with uuid aed67be5-d620-4f0c-83b2-f0154c001204 found in kube manager
2025-01-02 20:57:51,745 - INFO - Network policy with name default-policy-management:k8s-allowall found in agent
2025-01-02 20:57:51,745 - INFO - Network policy with name default-policy-management:k8s-allowall associated with default ks8
2025-01-02 20:57:51,745 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=aed67be5-d620-4f0c-83b2-f0154c001204
2025-01-02 20:57:51,786 - WARNING - Network policy with name ingress-on-service not found in agent
2025-01-02 20:57:53,787 - DEBUG - Requesting: http://10.0.0.30:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=aed67be5-d620-4f0c-83b2-f0154c001204
2025-01-02 20:57:53,819 - INFO - Network policy with name ingress-on-service found in agent
2025-01-02 20:57:53,819 - INFO - Network Policy ingress-on-service verification passed
2025-01-02 20:57:58,885 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:57:58,885 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:57:58,885 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:03,939 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:03,939 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:03,939 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:09,003 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:09,003 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:09,003 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:14,063 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:14,063 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:14,063 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:19,127 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:19,127 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:19,127 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:24,194 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:24,195 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:24,195 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:29,251 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:29,251 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:29,251 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:34,321 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:34,321 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:34,321 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:39,386 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:39,386 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:39,386 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:44,452 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:44,452 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:44,452 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:44,452 - INFO - As expected, responses not seen from podsHits : {'ctest-pod-16149351': 0, 'ctest-pod-01498961': 0}
2025-01-02 20:58:44,508 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 20:58:44,508 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 20:58:44,508 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 20:58:44,572 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-16149351
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 20:58:44,572 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 20:58:44,572 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: ctest-pod-16149351
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 20:58:44,572 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16149351': 1, 'ctest-pod-01498961': 1}
2025-01-02 20:58:49,624 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:49,624 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:49,624 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:54,679 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:54,679 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:54,679 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:59,743 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:59,743 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:58:59,743 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:04,811 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:04,811 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:04,811 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:09,872 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:09,872 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:09,872 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:14,942 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:14,943 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:14,943 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:20,020 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:20,020 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:20,020 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:25,083 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:25,083 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:25,083 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:30,144 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:30,145 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:30,145 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:35,191 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:35,191 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:35,191 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 20:59:35,191 - INFO - As expected, responses not seen from podsHits : {'ctest-pod-16149351': 0, 'ctest-pod-01498961': 0}
2025-01-02 20:59:37,242 - DEBUG - [Pod ctest-pod-09099449] Cmd: ping -c 3 10.47.255.247, Output: PING 10.47.255.247 (10.47.255.247): 56 data bytes
64 bytes from 10.47.255.247: seq=0 ttl=63 time=0.583 ms
64 bytes from 10.47.255.247: seq=1 ttl=63 time=0.077 ms
64 bytes from 10.47.255.247: seq=2 ttl=63 time=0.073 ms
--- 10.47.255.247 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.073/0.244/0.583 ms
2025-01-02 20:59:37,243 - INFO - Ping check to IP 10.47.255.247 from pod ctest-pod-09099449 with expectation True passed
2025-01-02 20:59:39,291 - DEBUG - [Pod ctest-pod-76539307] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes
64 bytes from 10.47.255.246: seq=0 ttl=63 time=0.354 ms
64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.071 ms
64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.086 ms
--- 10.47.255.246 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.071/0.170/0.354 ms
2025-01-02 20:59:39,291 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-76539307 with expectation True passed
2025-01-02 20:59:41,339 - DEBUG - [Pod ctest-pod-54167827] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes
64 bytes from 10.47.255.246: seq=0 ttl=63 time=0.402 ms
64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.068 ms
64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.066 ms
--- 10.47.255.246 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.066/0.178/0.402 ms
2025-01-02 20:59:41,339 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-54167827 with expectation True passed
2025-01-02 20:59:41,390 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.47.255.251 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.251 (10.47.255.251:80)
saving to '/dev/null'
null 100% |********************************| 19 0:00:00 ETA
'/dev/null' saved
2025-01-02 20:59:41,390 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.47.255.251 -O /dev/null -T 5 -t 1 passed
2025-01-02 20:59:41,391 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: Connecting to 10.47.255.251 (10.47.255.251:80)
saving to '/dev/null'
null 100% |********************************| 19 0:00:00 ETA
'/dev/null' saved
2025-01-02 20:59:41,391 - INFO - wget check of of http://10.47.255.251 from ctest-pod-09099449 passed
2025-01-02 20:59:41,454 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80)
saving to '/dev/null'
null 100% |********************************| 19 0:00:00 ETA
'/dev/null' saved
2025-01-02 20:59:41,454 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed
2025-01-02 20:59:41,454 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80)
saving to '/dev/null'
null 100% |********************************| 19 0:00:00 ETA
'/dev/null' saved
2025-01-02 20:59:41,454 - INFO - wget check of of http://10.47.255.250 from ctest-pod-09099449 passed
2025-01-02 20:59:46,507 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80)
wget: download timed out
2025-01-02 20:59:46,508 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 failed. Output :Connecting to 10.47.255.250 (10.47.255.250:80)
wget: download timed out
2025-01-02 20:59:46,508 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80)
wget: download timed out
2025-01-02 20:59:46,508 - INFO - wget check of of http://10.47.255.250 from ctest-pod-76539307 passed
2025-01-02 20:59:49,720 - WARNING - ApiException is caught (0)
Reason: Connection is already closed.
. Retrying...
2025-01-02 21:00:00,019 - DEBUG - [Pod ctest-pod-28354863] Cmd: wget http://10.47.255.251 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.251 (10.47.255.251:80)
wget: download timed out
2025-01-02 21:00:00,019 - DEBUG - [Pod Pod ctest-pod-28354863] Cmd wget http://10.47.255.251 -O /dev/null -T 5 -t 1 failed. Output :Connecting to 10.47.255.251 (10.47.255.251:80)
wget: download timed out
2025-01-02 21:00:00,019 - DEBUG - [Pod Pod ctest-pod-28354863] Cmd output: Connecting to 10.47.255.251 (10.47.255.251:80)
wget: download timed out
2025-01-02 21:00:00,019 - INFO - wget check of of http://10.47.255.251 from ctest-pod-28354863 passed
2025-01-02 21:00:00,034 - INFO - Updating Network Policy ingress-on-service
2025-01-02 21:00:05,127 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:05,127 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:05,127 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:10,184 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:10,185 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:10,185 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:15,257 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:15,257 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:15,258 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:20,324 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:20,324 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:20,324 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:25,387 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:25,388 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:25,388 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:30,451 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:30,452 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:30,452 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:35,508 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:35,508 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:35,508 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:40,559 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:40,559 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:40,559 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:45,621 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:45,622 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:45,622 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:50,697 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:50,697 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:50,697 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
wget: download timed out
2025-01-02 21:00:50,697 - INFO - As expected, responses not seen from podsHits : {'ctest-pod-16149351': 0, 'ctest-pod-01498961': 0}
2025-01-02 21:00:50,745 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-01498961
2025-01-02 21:00:50,745 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 21:00:50,745 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-01498961
2025-01-02 21:00:50,810 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:50,810 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 21:00:50,810 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:50,868 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:50,869 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 21:00:50,869 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:50,929 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:50,929 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 21:00:50,929 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:51,001 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:51,002 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 21:00:51,002 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:51,052 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:51,052 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 21:00:51,052 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:51,121 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-16149351
2025-01-02 21:00:51,121 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 21:00:51,121 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-16149351
2025-01-02 21:00:51,121 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16149351': 1, 'ctest-pod-01498961': 6}
2025-01-02 21:00:51,247 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:51,247 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 21:00:51,247 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: ctest-pod-01498961
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:51,318 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-01498961
2025-01-02 21:00:51,318 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 21:00:51,318 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
ctest-pod-01498961
2025-01-02 21:00:51,376 - DEBUG - [Pod ctest-pod-54167827] Cmd: wget http://10.98.227.106:80/ -O - -T 5 -t 1 , Output: ctest-pod-16149351
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:51,376 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd wget http://10.98.227.106:80/ -O - -T 5 -t 1 passed
2025-01-02 21:00:51,376 - DEBUG - [Pod Pod ctest-pod-54167827] Cmd output: ctest-pod-16149351
Connecting to 10.98.227.106:80 (10.98.227.106:80)
writing to stdout
- 100% |********************************| 19 0:00:00 ETA
written to stdout
2025-01-02 21:00:51,376 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16149351': 1, 'ctest-pod-01498961': 2}
2025-01-02 21:00:53,414 - DEBUG - [Pod ctest-pod-09099449] Cmd: ping -c 3 10.47.255.247, Output: PING 10.47.255.247 (10.47.255.247): 56 data bytes
64 bytes from 10.47.255.247: seq=0 ttl=63 time=0.884 ms
64 bytes from 10.47.255.247: seq=1 ttl=63 time=0.054 ms
64 bytes from 10.47.255.247: seq=2 ttl=63 time=0.070 ms
--- 10.47.255.247 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.054/0.336/0.884 ms
2025-01-02 21:00:53,415 - INFO - Ping check to IP 10.47.255.247 from pod ctest-pod-09099449 with expectation True passed
2025-01-02 21:00:55,487 - DEBUG - [Pod ctest-pod-76539307] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes
64 bytes from 10.47.255.246: seq=0 ttl=63 time=0.792 ms
64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.069 ms
64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.072 ms
--- 10.47.255.246 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.069/0.311/0.792 ms
2025-01-02 21:00:55,487 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-76539307 with expectation True passed
2025-01-02 21:00:57,546 - DEBUG - [Pod ctest-pod-54167827] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes
64 bytes from 10.47.255.246: seq=0 ttl=63 time=0.911 ms
64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.067 ms
64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.060 ms
--- 10.47.255.246 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.060/0.346/0.911 ms
2025-01-02 21:00:57,547 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-54167827 with expectation True passed
2025-01-02 21:00:57,614 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.47.255.251 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.251 (10.47.255.251:80)
saving to '/dev/null'
null 100% |********************************| 19 0:00:00 ETA
'/dev/null' saved
2025-01-02 21:00:57,615 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.47.255.251 -O /dev/null -T 5 -t 1 passed
2025-01-02 21:00:57,615 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: Connecting to 10.47.255.251 (10.47.255.251:80)
saving to '/dev/null'
null 100% |********************************| 19 0:00:00 ETA
'/dev/null' saved
2025-01-02 21:00:57,615 - INFO - wget check of of http://10.47.255.251 from ctest-pod-09099449 passed
2025-01-02 21:00:57,669 - DEBUG - [Pod ctest-pod-09099449] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80)
saving to '/dev/null'
null 100% |********************************| 19 0:00:00 ETA
'/dev/null' saved
2025-01-02 21:00:57,670 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed
2025-01-02 21:00:57,670 - DEBUG - [Pod Pod ctest-pod-09099449] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80)
saving to '/dev/null'
null 100% |********************************| 19 0:00:00 ETA
'/dev/null' saved
2025-01-02 21:00:57,670 - INFO - wget check of of http://10.47.255.250 from ctest-pod-09099449 passed
2025-01-02 21:01:02,725 - DEBUG - [Pod ctest-pod-76539307] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80)
wget: download timed out
2025-01-02 21:01:02,726 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 failed. Output :Connecting to 10.47.255.250 (10.47.255.250:80)
wget: download timed out
2025-01-02 21:01:02,726 - DEBUG - [Pod Pod ctest-pod-76539307] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80)
wget: download timed out
2025-01-02 21:01:02,726 - INFO - wget check of of http://10.47.255.250 from ctest-pod-76539307 passed
2025-01-02 21:01:02,777 - DEBUG - [Pod ctest-pod-28354863] Cmd: wget http://10.47.255.251 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.251 (10.47.255.251:80)
saving to '/dev/null'
null 100% |********************************| 19 0:00:00 ETA
'/dev/null' saved
2025-01-02 21:01:02,777 - DEBUG - [Pod Pod ctest-pod-28354863] Cmd wget http://10.47.255.251 -O /dev/null -T 5 -t 1 passed
2025-01-02 21:01:02,777 - DEBUG - [Pod Pod ctest-pod-28354863] Cmd output: Connecting to 10.47.255.251 (10.47.255.251:80)
saving to '/dev/null'
null 100% |********************************| 19 0:00:00 ETA
'/dev/null' saved
2025-01-02 21:01:02,777 - INFO - wget check of of http://10.47.255.251 from ctest-pod-28354863 passed
2025-01-02 21:01:02,778 - INFO - Deleting Network Policy : ingress-on-service
2025-01-02 21:01:02,779 - INFO - Deleting service : ctest-nginx-svc-49506625
2025-01-02 21:01:02,990 - DEBUG - No XMPP flaps were noticed during the test
2025-01-02 21:01:02,990 - INFO - END TEST : test_ingress_rule_on_pod_with_service : PASSED[0:03:14]
2025-01-02 21:01:02,990 - INFO - --------------------------------------------------------------------------------
2025-01-02 21:01:02,995 - INFO - Deleting pod ctest-new-default-76592153:ctest-pod-16149351
2025-01-02 21:01:02,996 - INFO - Deleting pod ctest-new-default-76592153:ctest-pod-01498961
2025-01-02 21:01:02,996 - INFO - Deleting pod ctest-non-default-30747418:ctest-pod-13703445
2025-01-02 21:01:02,997 - INFO - Deleting pod ctest-non-default-30747418:ctest-pod-90850542
2025-01-02 21:01:02,997 - INFO - Deleting pod ctest-new-default-76592153:ctest-pod-76539307
2025-01-02 21:01:02,997 - INFO - Deleting pod ctest-new-default-76592153:ctest-pod-09099449
2025-01-02 21:01:02,998 - INFO - Deleting pod ctest-non-default-30747418:ctest-pod-54167827
2025-01-02 21:01:02,998 - INFO - Deleting pod ctest-non-default-30747418:ctest-pod-28354863
2025-01-02 21:01:03,094 - WARNING - Pod uuid cf80041c-44ba-498f-bdc5-4dbacd3cf5e0 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:03,108 - WARNING - Pod uuid 095bf4aa-3875-4eae-8cfc-948692641635 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:03,109 - WARNING - Pod uuid f61165a0-d0e4-4919-bc5d-40ab352ff1da is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:03,109 - WARNING - Pod uuid 354b8260-2192-41f4-b049-dd41a21f6c37 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:03,110 - WARNING - Pod uuid 1f6b0126-165e-461b-83f1-456134a25450 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:03,111 - WARNING - Pod uuid 23f74e67-cc80-4c34-b25d-26844b4d339f is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:03,112 - WARNING - Pod uuid 54943c50-cb6c-4c21-b6d9-9f36cc4980bb is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:03,112 - WARNING - Pod uuid ee3d708c-6873-40c1-89af-b552b0bb2457 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:08,103 - WARNING - Pod uuid cf80041c-44ba-498f-bdc5-4dbacd3cf5e0 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:08,141 - DEBUG - Pod 354b8260-2192-41f4-b049-dd41a21f6c37 is not in agent 10.0.0.37 VM list
2025-01-02 21:01:08,143 - WARNING - Pod uuid 1f6b0126-165e-461b-83f1-456134a25450 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:08,146 - DEBUG - Pod 095bf4aa-3875-4eae-8cfc-948692641635 is not in agent 10.0.0.37 VM list
2025-01-02 21:01:08,148 - DEBUG - Pod 54943c50-cb6c-4c21-b6d9-9f36cc4980bb is not in agent 10.0.0.37 VM list
2025-01-02 21:01:08,155 - WARNING - Pod uuid 23f74e67-cc80-4c34-b25d-26844b4d339f is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:08,156 - WARNING - Pod uuid ee3d708c-6873-40c1-89af-b552b0bb2457 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:08,156 - WARNING - Pod uuid f61165a0-d0e4-4919-bc5d-40ab352ff1da is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:08,167 - DEBUG - VMI 2ed2d086-c94c-11ef-9f85-fa163e8eb557 is removed from agent 10.0.0.37
2025-01-02 21:01:08,167 - INFO - Verified that pod ctest-pod-13703445 is removed in agent
2025-01-02 21:01:08,170 - DEBUG - VMI 2f5400ca-c94c-11ef-9f85-fa163e8eb557 is removed from agent 10.0.0.37
2025-01-02 21:01:08,170 - INFO - Verified that pod ctest-pod-90850542 is removed in agent
2025-01-02 21:01:08,171 - DEBUG - VMI 2d9e6720-c94c-11ef-9f85-fa163e8eb557 is removed from agent 10.0.0.37
2025-01-02 21:01:08,171 - INFO - Verified that pod ctest-pod-16149351 is removed in agent
2025-01-02 21:01:13,111 - WARNING - Pod uuid cf80041c-44ba-498f-bdc5-4dbacd3cf5e0 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:13,155 - WARNING - Pod uuid 1f6b0126-165e-461b-83f1-456134a25450 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:13,173 - WARNING - Pod uuid 23f74e67-cc80-4c34-b25d-26844b4d339f is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:13,175 - WARNING - Pod uuid ee3d708c-6873-40c1-89af-b552b0bb2457 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:13,176 - WARNING - Pod uuid f61165a0-d0e4-4919-bc5d-40ab352ff1da is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:18,119 - WARNING - Pod uuid cf80041c-44ba-498f-bdc5-4dbacd3cf5e0 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:18,163 - WARNING - Pod uuid 1f6b0126-165e-461b-83f1-456134a25450 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:18,188 - WARNING - Pod uuid 23f74e67-cc80-4c34-b25d-26844b4d339f is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:18,190 - WARNING - Pod uuid f61165a0-d0e4-4919-bc5d-40ab352ff1da is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:18,190 - WARNING - Pod uuid ee3d708c-6873-40c1-89af-b552b0bb2457 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:23,127 - WARNING - Pod uuid cf80041c-44ba-498f-bdc5-4dbacd3cf5e0 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:23,172 - WARNING - Pod uuid 1f6b0126-165e-461b-83f1-456134a25450 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:23,208 - WARNING - Pod uuid 23f74e67-cc80-4c34-b25d-26844b4d339f is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:23,212 - WARNING - Pod uuid f61165a0-d0e4-4919-bc5d-40ab352ff1da is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:23,213 - WARNING - Pod uuid ee3d708c-6873-40c1-89af-b552b0bb2457 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:28,136 - WARNING - Pod uuid cf80041c-44ba-498f-bdc5-4dbacd3cf5e0 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:28,180 - WARNING - Pod uuid 1f6b0126-165e-461b-83f1-456134a25450 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:28,227 - WARNING - Pod uuid 23f74e67-cc80-4c34-b25d-26844b4d339f is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:28,230 - WARNING - Pod uuid ee3d708c-6873-40c1-89af-b552b0bb2457 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:28,230 - WARNING - Pod uuid f61165a0-d0e4-4919-bc5d-40ab352ff1da is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:33,155 - WARNING - Pod uuid cf80041c-44ba-498f-bdc5-4dbacd3cf5e0 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:33,190 - WARNING - Pod uuid 1f6b0126-165e-461b-83f1-456134a25450 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:33,244 - WARNING - Pod uuid f61165a0-d0e4-4919-bc5d-40ab352ff1da is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:33,245 - WARNING - Pod uuid 23f74e67-cc80-4c34-b25d-26844b4d339f is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:33,247 - WARNING - Pod uuid ee3d708c-6873-40c1-89af-b552b0bb2457 is still seen in agent 10.0.0.37 VM list
2025-01-02 21:01:38,162 - DEBUG - Pod cf80041c-44ba-498f-bdc5-4dbacd3cf5e0 is not in agent 10.0.0.37 VM list
2025-01-02 21:01:38,169 - DEBUG - VMI 305a3890-c94c-11ef-9f85-fa163e8eb557 is removed from agent 10.0.0.37
2025-01-02 21:01:38,169 - INFO - Verified that pod ctest-pod-09099449 is removed in agent
2025-01-02 21:01:38,207 - DEBUG - Pod 1f6b0126-165e-461b-83f1-456134a25450 is not in agent 10.0.0.37 VM list
2025-01-02 21:01:38,214 - DEBUG - VMI 2e20f474-c94c-11ef-9f85-fa163e8eb557 is removed from agent 10.0.0.37
2025-01-02 21:01:38,214 - INFO - Verified that pod ctest-pod-01498961 is removed in agent
2025-01-02 21:01:38,259 - DEBUG - Pod ee3d708c-6873-40c1-89af-b552b0bb2457 is not in agent 10.0.0.37 VM list
2025-01-02 21:01:38,261 - DEBUG - Pod 23f74e67-cc80-4c34-b25d-26844b4d339f is not in agent 10.0.0.37 VM list
2025-01-02 21:01:38,262 - DEBUG - Pod f61165a0-d0e4-4919-bc5d-40ab352ff1da is not in agent 10.0.0.37 VM list
2025-01-02 21:01:38,275 - DEBUG - VMI 30d41dcc-c94c-11ef-9f85-fa163e8eb557 is removed from agent 10.0.0.37
2025-01-02 21:01:38,275 - INFO - Verified that pod ctest-pod-54167827 is removed in agent
2025-01-02 21:01:38,275 - DEBUG - VMI 314b9b4a-c94c-11ef-9f85-fa163e8eb557 is removed from agent 10.0.0.37
2025-01-02 21:01:38,275 - INFO - Verified that pod ctest-pod-28354863 is removed in agent
2025-01-02 21:01:38,276 - DEBUG - VMI 2fe85c20-c94c-11ef-9f85-fa163e8eb557 is removed from agent 10.0.0.37
2025-01-02 21:01:38,276 - INFO - Verified that pod ctest-pod-76539307 is removed in agent
2025-01-02 21:01:38,276 - INFO - Deleting namespace ctest-non-default-30747418
2025-01-02 21:01:38,298 - DEBUG - Namespace ctest-non-default-30747418 still in k8s
2025-01-02 21:01:40,302 - DEBUG - Namespace ctest-non-default-30747418 still in k8s
2025-01-02 21:01:42,308 - DEBUG - Namespace ctest-non-default-30747418 still in k8s
2025-01-02 21:01:44,311 - DEBUG - Namespace ctest-non-default-30747418 is not in k8s
2025-01-02 21:01:44,311 - DEBUG - No need to check project deletion in contrail-api
2025-01-02 21:01:44,311 - INFO - Deleting namespace ctest-new-default-76592153
2025-01-02 21:01:44,323 - DEBUG - Namespace ctest-new-default-76592153 still in k8s
2025-01-02 21:01:46,329 - DEBUG - Namespace ctest-new-default-76592153 still in k8s
2025-01-02 21:01:48,335 - DEBUG - Namespace ctest-new-default-76592153 still in k8s
2025-01-02 21:01:50,338 - DEBUG - Namespace ctest-new-default-76592153 is not in k8s
2025-01-02 21:01:50,339 - DEBUG - No need to check project deletion in contrail-api