2026-02-09 23:53:26,433 - INFO - Using existing project ['default-domain', 'k8s-default'](94ad159a-0929-4b4d-9eed-43fd79ff7d63) 2026-02-09 23:53:26,449 - DEBUG - Namespace ctest-new-default-93935537 not present 2026-02-09 23:53:26,459 - INFO - Created namespace ctest-new-default-93935537 2026-02-09 23:53:29,463 - DEBUG - Namespace ctest-non-default-93377265 not present 2026-02-09 23:53:29,476 - INFO - Created namespace ctest-non-default-93377265 2026-02-09 23:53:32,496 - DEBUG - Pod ctest-pod-16592486 not present 2026-02-09 23:53:32,496 - INFO - Creating Pod ctest-pod-16592486 2026-02-09 23:53:32,517 - DEBUG - Pod : ctest-pod-16592486 UUID is a1b08692-93c4-4e4c-a47d-1d59098790b1 2026-02-09 23:53:32,526 - DEBUG - Pod ctest-pod-81257232 not present 2026-02-09 23:53:32,527 - INFO - Creating Pod ctest-pod-81257232 2026-02-09 23:53:32,585 - DEBUG - Pod : ctest-pod-81257232 UUID is f2e1d5f9-7af1-484d-afb9-ba694e87233c 2026-02-09 23:53:32,587 - DEBUG - Pod ctest-pod-57049911 not present 2026-02-09 23:53:32,588 - INFO - Creating Pod ctest-pod-57049911 2026-02-09 23:53:32,631 - DEBUG - Pod : ctest-pod-57049911 UUID is 857405c8-f8b5-4c1b-82c2-3e124140950e 2026-02-09 23:53:32,638 - DEBUG - Pod ctest-pod-88393572 not present 2026-02-09 23:53:32,639 - INFO - Creating Pod ctest-pod-88393572 2026-02-09 23:53:32,668 - DEBUG - Pod : ctest-pod-88393572 UUID is a3ac4e60-f5fc-4ee8-a8dc-07d32a30b122 2026-02-09 23:53:32,671 - DEBUG - Pod ctest-pod-16862472 not present 2026-02-09 23:53:32,671 - INFO - Creating Pod ctest-pod-16862472 2026-02-09 23:53:32,715 - DEBUG - Pod : ctest-pod-16862472 UUID is 79af9e18-97d8-44e5-a366-cedc14826c58 2026-02-09 23:53:32,718 - DEBUG - Pod ctest-pod-07128938 not present 2026-02-09 23:53:32,718 - INFO - Creating Pod ctest-pod-07128938 2026-02-09 23:53:32,746 - DEBUG - Pod : ctest-pod-07128938 UUID is d8466479-4610-4556-ae22-8c74a428441b 2026-02-09 23:53:32,748 - DEBUG - Pod ctest-pod-02162332 not present 2026-02-09 23:53:32,748 - INFO - Creating Pod ctest-pod-02162332 2026-02-09 23:53:32,768 - DEBUG - Pod : ctest-pod-02162332 UUID is 6b3d91e9-043e-47c6-95e5-984c51727950 2026-02-09 23:53:32,770 - DEBUG - Pod ctest-pod-17903959 not present 2026-02-09 23:53:32,771 - INFO - Creating Pod ctest-pod-17903959 2026-02-09 23:53:32,803 - DEBUG - Pod : ctest-pod-17903959 UUID is 95ec114f-0a69-4c49-a120-7ac3d67279cd 2026-02-09 23:53:33,033 - INFO - Project default-domain:k8s-ctest-new-default-93935537 for Namespace ctest-new-default-93935537 is seen in contrail-api 2026-02-09 23:53:33,033 - DEBUG - Project uuid in Contrail is ee74673c-87cd-47f7-a780-c8845b02d272 2026-02-09 23:53:33,200 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_NamespaceDatabaseList?namespace_uuid=0bc87f5c-67b7-4e8a-8e58-6a611abc9b64 2026-02-09 23:53:33,204 - INFO - Namespace ctest-new-default-93935537 with uuid 0bc87f5c-67b7-4e8a-8e58-6a611abc9b64 found in kube manager 2026-02-09 23:53:33,208 - INFO - Namespace ctest-new-default-93935537 verification passed 2026-02-09 23:53:33,413 - INFO - Project default-domain:k8s-ctest-non-default-93377265 for Namespace ctest-non-default-93377265 is seen in contrail-api 2026-02-09 23:53:33,413 - DEBUG - Project uuid in Contrail is ac1acd40-b82d-4f37-bd71-c31d7e937533 2026-02-09 23:53:33,413 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_NamespaceDatabaseList?namespace_uuid=a7b69c45-9ee5-49af-a1ad-af4d7a808af2 2026-02-09 23:53:33,417 - INFO - Namespace ctest-non-default-93377265 with uuid a7b69c45-9ee5-49af-a1ad-af4d7a808af2 found in kube manager 2026-02-09 23:53:33,417 - INFO - Namespace ctest-non-default-93377265 verification passed 2026-02-09 23:53:33,423 - DEBUG - Pod ctest-pod-16592486 not in running state.Currently in Pending 2026-02-09 23:53:38,431 - DEBUG - Pod ctest-pod-16592486 not in running state.Currently in Pending 2026-02-09 23:53:43,438 - DEBUG - Pod ctest-pod-16592486 not in running state.Currently in Pending 2026-02-09 23:53:48,444 - INFO - Pod ctest-pod-16592486 is in running state.Got IP 10.47.255.251 2026-02-09 23:53:48,494 - DEBUG - Pod ctest-pod-16592486 has vmi 8b4b5438-0612-11f1-9e0b-fa163e567dec 2026-02-09 23:53:48,494 - INFO - Verified pod ctest-pod-16592486 in contrail-api 2026-02-09 23:53:48,507 - DEBUG - VMI 8b4b5438-0612-11f1-9e0b-fa163e567dec is active in agent 10.0.0.129 2026-02-09 23:53:48,507 - INFO - Verified Pod ctest-pod-16592486 in agent 10.0.0.129 2026-02-09 23:53:48,507 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=a1b08692-93c4-4e4c-a47d-1d59098790b1 2026-02-09 23:53:48,510 - INFO - Pod ctest-pod-16592486 with uuid a1b08692-93c4-4e4c-a47d-1d59098790b1 found in kube manager 2026-02-09 23:53:48,511 - INFO - Pod ctest-pod-16592486 verification passed 2026-02-09 23:53:48,517 - INFO - Pod ctest-pod-81257232 is in running state.Got IP 10.47.255.250 2026-02-09 23:53:48,526 - DEBUG - Pod ctest-pod-81257232 has vmi 8bbf1012-0612-11f1-9e0b-fa163e567dec 2026-02-09 23:53:48,526 - INFO - Verified pod ctest-pod-81257232 in contrail-api 2026-02-09 23:53:48,537 - DEBUG - VMI 8bbf1012-0612-11f1-9e0b-fa163e567dec is active in agent 10.0.0.129 2026-02-09 23:53:48,537 - INFO - Verified Pod ctest-pod-81257232 in agent 10.0.0.129 2026-02-09 23:53:48,537 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=f2e1d5f9-7af1-484d-afb9-ba694e87233c 2026-02-09 23:53:48,540 - INFO - Pod ctest-pod-81257232 with uuid f2e1d5f9-7af1-484d-afb9-ba694e87233c found in kube manager 2026-02-09 23:53:48,540 - INFO - Pod ctest-pod-81257232 verification passed 2026-02-09 23:53:48,546 - INFO - Pod ctest-pod-57049911 is in running state.Got IP 10.47.255.249 2026-02-09 23:53:48,555 - DEBUG - Pod ctest-pod-57049911 has vmi 8c521178-0612-11f1-9e0b-fa163e567dec 2026-02-09 23:53:48,555 - INFO - Verified pod ctest-pod-57049911 in contrail-api 2026-02-09 23:53:48,565 - DEBUG - VMI 8c521178-0612-11f1-9e0b-fa163e567dec is active in agent 10.0.0.129 2026-02-09 23:53:48,565 - INFO - Verified Pod ctest-pod-57049911 in agent 10.0.0.129 2026-02-09 23:53:48,565 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=857405c8-f8b5-4c1b-82c2-3e124140950e 2026-02-09 23:53:48,568 - INFO - Pod ctest-pod-57049911 with uuid 857405c8-f8b5-4c1b-82c2-3e124140950e found in kube manager 2026-02-09 23:53:48,568 - INFO - Pod ctest-pod-57049911 verification passed 2026-02-09 23:53:48,573 - INFO - Pod ctest-pod-88393572 is in running state.Got IP 10.47.255.248 2026-02-09 23:53:48,624 - DEBUG - Pod ctest-pod-88393572 has vmi 8cc276f2-0612-11f1-9e0b-fa163e567dec 2026-02-09 23:53:48,625 - INFO - Verified pod ctest-pod-88393572 in contrail-api 2026-02-09 23:53:48,636 - DEBUG - VMI 8cc276f2-0612-11f1-9e0b-fa163e567dec is active in agent 10.0.0.129 2026-02-09 23:53:48,636 - INFO - Verified Pod ctest-pod-88393572 in agent 10.0.0.129 2026-02-09 23:53:48,636 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=a3ac4e60-f5fc-4ee8-a8dc-07d32a30b122 2026-02-09 23:53:48,639 - INFO - Pod ctest-pod-88393572 with uuid a3ac4e60-f5fc-4ee8-a8dc-07d32a30b122 found in kube manager 2026-02-09 23:53:48,639 - INFO - Pod ctest-pod-88393572 verification passed 2026-02-09 23:53:48,646 - DEBUG - Pod ctest-pod-16862472 not in running state.Currently in Pending 2026-02-09 23:53:53,652 - INFO - Pod ctest-pod-16862472 is in running state.Got IP 10.47.255.247 2026-02-09 23:53:53,663 - DEBUG - Pod ctest-pod-16862472 has vmi 8d35a0e6-0612-11f1-9e0b-fa163e567dec 2026-02-09 23:53:53,663 - INFO - Verified pod ctest-pod-16862472 in contrail-api 2026-02-09 23:53:53,674 - DEBUG - VMI 8d35a0e6-0612-11f1-9e0b-fa163e567dec is active in agent 10.0.0.129 2026-02-09 23:53:53,674 - INFO - Verified Pod ctest-pod-16862472 in agent 10.0.0.129 2026-02-09 23:53:53,674 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=79af9e18-97d8-44e5-a366-cedc14826c58 2026-02-09 23:53:53,678 - INFO - Pod ctest-pod-16862472 with uuid 79af9e18-97d8-44e5-a366-cedc14826c58 found in kube manager 2026-02-09 23:53:53,678 - INFO - Pod ctest-pod-16862472 verification passed 2026-02-09 23:53:53,684 - INFO - Pod ctest-pod-02162332 is in running state.Got IP 10.47.255.245 2026-02-09 23:53:53,736 - DEBUG - Pod ctest-pod-02162332 has vmi 8dfd3c82-0612-11f1-9e0b-fa163e567dec 2026-02-09 23:53:53,737 - INFO - Verified pod ctest-pod-02162332 in contrail-api 2026-02-09 23:53:53,747 - DEBUG - VMI 8dfd3c82-0612-11f1-9e0b-fa163e567dec is active in agent 10.0.0.129 2026-02-09 23:53:53,747 - INFO - Verified Pod ctest-pod-02162332 in agent 10.0.0.129 2026-02-09 23:53:53,747 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=6b3d91e9-043e-47c6-95e5-984c51727950 2026-02-09 23:53:53,751 - INFO - Pod ctest-pod-02162332 with uuid 6b3d91e9-043e-47c6-95e5-984c51727950 found in kube manager 2026-02-09 23:53:53,751 - INFO - Pod ctest-pod-02162332 verification passed 2026-02-09 23:53:53,756 - INFO - Pod ctest-pod-07128938 is in running state.Got IP 10.47.255.246 2026-02-09 23:53:53,806 - DEBUG - Pod ctest-pod-07128938 has vmi 8d976420-0612-11f1-9e0b-fa163e567dec 2026-02-09 23:53:53,807 - INFO - Verified pod ctest-pod-07128938 in contrail-api 2026-02-09 23:53:53,820 - DEBUG - VMI 8d976420-0612-11f1-9e0b-fa163e567dec is active in agent 10.0.0.129 2026-02-09 23:53:53,820 - INFO - Verified Pod ctest-pod-07128938 in agent 10.0.0.129 2026-02-09 23:53:53,820 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=d8466479-4610-4556-ae22-8c74a428441b 2026-02-09 23:53:53,824 - INFO - Pod ctest-pod-07128938 with uuid d8466479-4610-4556-ae22-8c74a428441b found in kube manager 2026-02-09 23:53:53,824 - INFO - Pod ctest-pod-07128938 verification passed 2026-02-09 23:53:53,831 - INFO - Pod ctest-pod-17903959 is in running state.Got IP 10.47.255.244 2026-02-09 23:53:53,844 - DEBUG - Pod ctest-pod-17903959 has vmi 8e61348a-0612-11f1-9e0b-fa163e567dec 2026-02-09 23:53:53,844 - INFO - Verified pod ctest-pod-17903959 in contrail-api 2026-02-09 23:53:53,855 - DEBUG - VMI 8e61348a-0612-11f1-9e0b-fa163e567dec is active in agent 10.0.0.129 2026-02-09 23:53:53,855 - INFO - Verified Pod ctest-pod-17903959 in agent 10.0.0.129 2026-02-09 23:53:53,855 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=95ec114f-0a69-4c49-a120-7ac3d67279cd 2026-02-09 23:53:53,859 - INFO - Pod ctest-pod-17903959 with uuid 95ec114f-0a69-4c49-a120-7ac3d67279cd found in kube manager 2026-02-09 23:53:53,859 - INFO - Pod ctest-pod-17903959 verification passed 2026-02-09 23:53:53,859 - INFO - ================================================================================ 2026-02-09 23:53:53,859 - INFO - STARTING TEST : test_ingress_rule_on_pod_with_service 2026-02-09 23:53:53,859 - INFO - TEST DESCRIPTION : Verify that ingress rules on a pod work fine when pods are accessed via a service Steps: 1. Create a service in namespace 1 over a pod which are endpoints of a service 2. Create a network policy to allow 1 pod of namespace 1 as ingress. 3. Verify that service respond correctly to the specified pod in the policy and service do not respond to anyone else. 4. Extend the policy and add the namespaceselector as namespace 2. 5. Verify that service become accessible to the pods of namespace 2. 2026-02-09 23:53:54,040 - DEBUG - Nothing to compare xmpp stats {'10.0.0.129': {'10.0.0.22': '0'}} with 2026-02-09 23:53:54,040 - INFO - Initial checks done. Running the testcase now 2026-02-09 23:53:54,040 - INFO - 2026-02-09 23:53:54,094 - DEBUG - [Pod ctest-pod-16592486] Cmd: echo ctest-pod-16592486 > /usr/share/nginx/html/index.html, Output: 2026-02-09 23:53:54,151 - DEBUG - [Pod ctest-pod-81257232] Cmd: echo ctest-pod-81257232 > /usr/share/nginx/html/index.html, Output: 2026-02-09 23:53:54,205 - DEBUG - [Pod ctest-pod-57049911] Cmd: echo ctest-pod-57049911 > /usr/share/nginx/html/index.html, Output: 2026-02-09 23:53:54,262 - DEBUG - [Pod ctest-pod-88393572] Cmd: echo ctest-pod-88393572 > /usr/share/nginx/html/index.html, Output: 2026-02-09 23:53:54,265 - DEBUG - Service ctest-nginx-svc-04856776 not present 2026-02-09 23:53:54,265 - INFO - Creating service ctest-nginx-svc-04856776 2026-02-09 23:53:54,283 - INFO - Created Service ctest-nginx-svc-04856776 2026-02-09 23:53:54,303 - WARNING - Service UUID 55c0af64-c11a-4789-ad6c-5a7ee403c76b not yet found in contrail-api 2026-02-09 23:53:55,321 - INFO - Validated that Service ctest-nginx-svc-04856776 is seen in contrail-api 2026-02-09 23:53:55,321 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=55c0af64-c11a-4789-ad6c-5a7ee403c76b 2026-02-09 23:53:55,329 - INFO - Service ctest-nginx-svc-04856776 with uuid 55c0af64-c11a-4789-ad6c-5a7ee403c76b found in kube manager 2026-02-09 23:53:55,329 - INFO - Service ctest-nginx-svc-04856776 verification passed 2026-02-09 23:53:55,390 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: ctest-pod-16592486 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:53:55,390 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 passed 2026-02-09 23:53:55,390 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: ctest-pod-16592486 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:53:55,441 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: ctest-pod-16592486 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:53:55,441 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 passed 2026-02-09 23:53:55,441 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: ctest-pod-16592486 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:53:55,494 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-16592486 2026-02-09 23:53:55,494 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 passed 2026-02-09 23:53:55,495 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-16592486 2026-02-09 23:53:55,549 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: ctest-pod-81257232 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:53:55,549 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 passed 2026-02-09 23:53:55,549 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: ctest-pod-81257232 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:53:55,549 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16592486': 3, 'ctest-pod-81257232': 1} 2026-02-09 23:53:55,602 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-81257232 2026-02-09 23:53:55,602 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 passed 2026-02-09 23:53:55,602 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout ctest-pod-81257232 2026-02-09 23:53:55,658 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: ctest-pod-16592486 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:53:55,658 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 passed 2026-02-09 23:53:55,658 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: ctest-pod-16592486 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:53:55,658 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16592486': 1, 'ctest-pod-81257232': 1} 2026-02-09 23:53:55,660 - DEBUG - Network policy ingress-on-service not present 2026-02-09 23:53:55,661 - INFO - Creating Network Policy ingress-on-service 2026-02-09 23:53:55,677 - INFO - Network policy found in k8s 2026-02-09 23:53:55,677 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=48283a98-ee03-4431-a619-87dedc85b049 2026-02-09 23:53:55,681 - INFO - Network Policy ingress-on-service with uuid 48283a98-ee03-4431-a619-87dedc85b049 found in kube manager 2026-02-09 23:53:55,705 - INFO - Network policy with name default-policy-management:k8s-allowall found in agent 2026-02-09 23:53:55,705 - INFO - Network policy with name default-policy-management:k8s-allowall associated with default ks8 2026-02-09 23:53:55,705 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=48283a98-ee03-4431-a619-87dedc85b049 2026-02-09 23:53:55,725 - WARNING - Network policy with name ingress-on-service not found in agent 2026-02-09 23:53:57,725 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_NetworkPolicyDatabaseList?network_policy_uuid=48283a98-ee03-4431-a619-87dedc85b049 2026-02-09 23:53:57,749 - INFO - Network policy with name ingress-on-service found in agent 2026-02-09 23:53:57,749 - INFO - Network Policy ingress-on-service verification passed 2026-02-09 23:54:02,791 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:02,791 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:02,791 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:07,858 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:07,858 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:07,858 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:08,384 - WARNING - ApiException is caught (0) Reason: Connection is already closed. . Retrying... 2026-02-09 23:54:18,437 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:18,438 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:18,438 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:23,499 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:23,499 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:23,499 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:28,550 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:28,550 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:28,550 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:33,603 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:33,603 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:33,603 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:38,667 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:38,667 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:38,667 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:43,723 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:43,723 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:43,723 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:48,773 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:48,774 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:48,774 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:53,823 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:53,823 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:53,823 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:53,823 - INFO - As expected, responses not seen from podsHits : {'ctest-pod-16592486': 0, 'ctest-pod-81257232': 0} 2026-02-09 23:54:53,888 - DEBUG - [Pod ctest-pod-07128938] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: ctest-pod-16592486 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:54:53,888 - DEBUG - [Pod Pod ctest-pod-07128938] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 passed 2026-02-09 23:54:53,888 - DEBUG - [Pod Pod ctest-pod-07128938] Cmd output: ctest-pod-16592486 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:54:53,949 - DEBUG - [Pod ctest-pod-07128938] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: ctest-pod-81257232 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:54:53,949 - DEBUG - [Pod Pod ctest-pod-07128938] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 passed 2026-02-09 23:54:53,949 - DEBUG - [Pod Pod ctest-pod-07128938] Cmd output: ctest-pod-81257232 Connecting to 10.96.208.225:80 (10.96.208.225:80) writing to stdout - 100% |********************************| 19 0:00:00 ETA written to stdout 2026-02-09 23:54:53,949 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-pod-16592486': 1, 'ctest-pod-81257232': 1} 2026-02-09 23:54:58,994 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:58,994 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:54:58,994 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:04,080 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:04,080 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:04,080 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:09,127 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:09,127 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:09,127 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:14,178 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:14,178 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:14,178 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:19,240 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:19,240 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:19,240 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:24,295 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:24,295 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:24,295 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:29,347 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:29,347 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:29,347 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:34,399 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:34,399 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:34,400 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:39,450 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:39,451 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:39,451 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:44,507 - DEBUG - [Pod ctest-pod-02162332] Cmd: wget http://10.96.208.225:80/ -O - -T 5 -t 1 , Output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:44,507 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd wget http://10.96.208.225:80/ -O - -T 5 -t 1 failed. Output :Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:44,507 - DEBUG - [Pod Pod ctest-pod-02162332] Cmd output: Connecting to 10.96.208.225:80 (10.96.208.225:80) wget: download timed out 2026-02-09 23:55:44,507 - INFO - As expected, responses not seen from podsHits : {'ctest-pod-16592486': 0, 'ctest-pod-81257232': 0} 2026-02-09 23:55:46,558 - DEBUG - [Pod ctest-pod-07128938] Cmd: ping -c 3 10.47.255.247, Output: PING 10.47.255.247 (10.47.255.247): 56 data bytes 64 bytes from 10.47.255.247: seq=0 ttl=63 time=1.003 ms 64 bytes from 10.47.255.247: seq=1 ttl=63 time=0.078 ms 64 bytes from 10.47.255.247: seq=2 ttl=63 time=0.093 ms --- 10.47.255.247 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.078/0.391/1.003 ms 2026-02-09 23:55:46,558 - INFO - Ping check to IP 10.47.255.247 from pod ctest-pod-07128938 with expectation True passed 2026-02-09 23:55:48,610 - DEBUG - [Pod ctest-pod-16862472] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes 64 bytes from 10.47.255.246: seq=0 ttl=63 time=1.484 ms 64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.078 ms 64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.075 ms --- 10.47.255.246 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.075/0.545/1.484 ms 2026-02-09 23:55:48,610 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-16862472 with expectation True passed 2026-02-09 23:55:50,672 - DEBUG - [Pod ctest-pod-02162332] Cmd: ping -c 3 10.47.255.246, Output: PING 10.47.255.246 (10.47.255.246): 56 data bytes 64 bytes from 10.47.255.246: seq=0 ttl=63 time=0.891 ms 64 bytes from 10.47.255.246: seq=1 ttl=63 time=0.080 ms 64 bytes from 10.47.255.246: seq=2 ttl=63 time=0.093 ms --- 10.47.255.246 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.080/0.354/0.891 ms 2026-02-09 23:55:50,672 - INFO - Ping check to IP 10.47.255.246 from pod ctest-pod-02162332 with expectation True passed 2026-02-09 23:55:50,726 - DEBUG - [Pod ctest-pod-07128938] Cmd: wget http://10.47.255.251 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.251 (10.47.255.251:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:50,726 - DEBUG - [Pod Pod ctest-pod-07128938] Cmd wget http://10.47.255.251 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:55:50,726 - DEBUG - [Pod Pod ctest-pod-07128938] Cmd output: Connecting to 10.47.255.251 (10.47.255.251:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:50,726 - INFO - wget check of of http://10.47.255.251 from ctest-pod-07128938 passed 2026-02-09 23:55:50,779 - DEBUG - [Pod ctest-pod-07128938] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:50,779 - DEBUG - [Pod Pod ctest-pod-07128938] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:55:50,779 - DEBUG - [Pod Pod ctest-pod-07128938] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:50,779 - INFO - wget check of of http://10.47.255.250 from ctest-pod-07128938 passed 2026-02-09 23:55:50,831 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:50,831 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:55:50,831 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:50,831 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:55:52,885 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:52,885 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:55:52,885 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:52,885 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:55:54,937 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:54,937 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:55:54,937 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:54,937 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:55:56,992 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:56,992 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:55:56,993 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:56,993 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:55:59,050 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:59,050 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:55:59,050 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:55:59,050 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:56:01,105 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:01,105 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:56:01,105 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:01,105 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:56:03,160 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:03,160 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:56:03,160 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:03,160 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:56:05,228 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:05,228 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:56:05,228 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:05,228 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:56:07,283 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:07,283 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:56:07,283 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:07,283 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:56:09,336 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:09,336 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:56:09,336 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:09,336 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:56:11,393 - DEBUG - [Pod ctest-pod-16862472] Cmd: wget http://10.47.255.250 -O /dev/null -T 5 -t 1 , Output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:11,393 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd wget http://10.47.255.250 -O /dev/null -T 5 -t 1 passed 2026-02-09 23:56:11,393 - DEBUG - [Pod Pod ctest-pod-16862472] Cmd output: Connecting to 10.47.255.250 (10.47.255.250:80) saving to '/dev/null' null 100% |********************************| 19 0:00:00 ETA '/dev/null' saved 2026-02-09 23:56:11,393 - WARNING - wget check of of http://10.47.255.250 from ctest-pod-16862472 failed 2026-02-09 23:56:11,398 - INFO - Deleting Network Policy : ingress-on-service 2026-02-09 23:56:11,398 - INFO - Deleting service : ctest-nginx-svc-04856776 2026-02-09 23:56:11,615 - ERROR - AssertionError Python 3.9.21: /usr/bin/python3 Mon Feb 9 23:56:11 2026 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/k8s_scripts/test_policy.py in test_ingress_rule_on_pod_with_service(self=) 2859 assert self.validate_wget(self.client2_pod_ns1, url1) 2860 assert self.validate_wget(self.client2_pod_ns1, url2) 2861 assert self.validate_wget(self.client1_pod_ns1, url2, expectation=False) 2862 assert self.validate_wget(self.client2_pod_ns2, url1, expectation=False) 2863 # Updating the policy and adding the namespace selector self = self.validate_wget = .deco_retry....ice[ci_k8s_sanity,k8s_sanity] id=0x7f8fbff61af0>> self.client1_pod_ns1 = url2 = 'http://10.47.255.250' expectation undefined AssertionError: __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = () with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/k8s_scripts/test_policy.py", line 2861, in test_ingress_rule_on_pod_with_service assert self.validate_wget(self.client1_pod_ns1, url2, expectation=False) AssertionError 2026-02-09 23:56:11,621 - DEBUG - No XMPP flaps were noticed during the test 2026-02-09 23:56:11,622 - INFO -
2026-02-09 23:56:11,622 - INFO - END TEST : test_ingress_rule_on_pod_with_service : FAILED[0:02:18]
2026-02-09 23:56:11,622 - INFO - -------------------------------------------------------------------------------- 2026-02-09 23:56:11,626 - INFO - Deleting pod ctest-new-default-93935537:ctest-pod-16592486 2026-02-09 23:56:11,626 - INFO - Deleting pod ctest-new-default-93935537:ctest-pod-81257232 2026-02-09 23:56:11,626 - INFO - Deleting pod ctest-non-default-93377265:ctest-pod-57049911 2026-02-09 23:56:11,627 - INFO - Deleting pod ctest-non-default-93377265:ctest-pod-88393572 2026-02-09 23:56:11,627 - INFO - Deleting pod ctest-new-default-93935537:ctest-pod-16862472 2026-02-09 23:56:11,627 - INFO - Deleting pod ctest-new-default-93935537:ctest-pod-07128938 2026-02-09 23:56:11,628 - INFO - Deleting pod ctest-non-default-93377265:ctest-pod-02162332 2026-02-09 23:56:11,628 - INFO - Deleting pod ctest-non-default-93377265:ctest-pod-17903959 2026-02-09 23:56:11,694 - WARNING - Pod uuid 6b3d91e9-043e-47c6-95e5-984c51727950 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:11,715 - WARNING - Pod uuid 95ec114f-0a69-4c49-a120-7ac3d67279cd is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:11,716 - WARNING - Pod uuid a1b08692-93c4-4e4c-a47d-1d59098790b1 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:11,716 - WARNING - Pod uuid 79af9e18-97d8-44e5-a366-cedc14826c58 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:11,723 - WARNING - Pod uuid a3ac4e60-f5fc-4ee8-a8dc-07d32a30b122 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:11,727 - WARNING - Pod uuid 857405c8-f8b5-4c1b-82c2-3e124140950e is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:11,728 - WARNING - Pod uuid f2e1d5f9-7af1-484d-afb9-ba694e87233c is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:11,729 - WARNING - Pod uuid d8466479-4610-4556-ae22-8c74a428441b is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:16,701 - WARNING - Pod uuid 6b3d91e9-043e-47c6-95e5-984c51727950 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:16,726 - WARNING - Pod uuid 95ec114f-0a69-4c49-a120-7ac3d67279cd is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:16,728 - WARNING - Pod uuid 79af9e18-97d8-44e5-a366-cedc14826c58 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:16,729 - DEBUG - Pod a1b08692-93c4-4e4c-a47d-1d59098790b1 is not in agent 10.0.0.129 VM list 2026-02-09 23:56:16,733 - DEBUG - Pod a3ac4e60-f5fc-4ee8-a8dc-07d32a30b122 is not in agent 10.0.0.129 VM list 2026-02-09 23:56:16,743 - DEBUG - VMI 8b4b5438-0612-11f1-9e0b-fa163e567dec is removed from agent 10.0.0.129 2026-02-09 23:56:16,743 - INFO - Verified that pod ctest-pod-16592486 is removed in agent 2026-02-09 23:56:16,745 - DEBUG - VMI 8cc276f2-0612-11f1-9e0b-fa163e567dec is removed from agent 10.0.0.129 2026-02-09 23:56:16,745 - INFO - Verified that pod ctest-pod-88393572 is removed in agent 2026-02-09 23:56:16,745 - WARNING - Pod uuid d8466479-4610-4556-ae22-8c74a428441b is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:16,745 - DEBUG - Pod f2e1d5f9-7af1-484d-afb9-ba694e87233c is not in agent 10.0.0.129 VM list 2026-02-09 23:56:16,746 - DEBUG - Pod 857405c8-f8b5-4c1b-82c2-3e124140950e is not in agent 10.0.0.129 VM list 2026-02-09 23:56:16,754 - DEBUG - VMI 8bbf1012-0612-11f1-9e0b-fa163e567dec is removed from agent 10.0.0.129 2026-02-09 23:56:16,754 - INFO - Verified that pod ctest-pod-81257232 is removed in agent 2026-02-09 23:56:16,754 - DEBUG - VMI 8c521178-0612-11f1-9e0b-fa163e567dec is removed from agent 10.0.0.129 2026-02-09 23:56:16,754 - INFO - Verified that pod ctest-pod-57049911 is removed in agent 2026-02-09 23:56:21,708 - WARNING - Pod uuid 6b3d91e9-043e-47c6-95e5-984c51727950 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:21,736 - WARNING - Pod uuid 95ec114f-0a69-4c49-a120-7ac3d67279cd is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:21,737 - WARNING - Pod uuid 79af9e18-97d8-44e5-a366-cedc14826c58 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:21,751 - WARNING - Pod uuid d8466479-4610-4556-ae22-8c74a428441b is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:26,714 - WARNING - Pod uuid 6b3d91e9-043e-47c6-95e5-984c51727950 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:26,746 - WARNING - Pod uuid 79af9e18-97d8-44e5-a366-cedc14826c58 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:26,746 - WARNING - Pod uuid 95ec114f-0a69-4c49-a120-7ac3d67279cd is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:26,757 - WARNING - Pod uuid d8466479-4610-4556-ae22-8c74a428441b is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:31,722 - WARNING - Pod uuid 6b3d91e9-043e-47c6-95e5-984c51727950 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:31,755 - WARNING - Pod uuid 95ec114f-0a69-4c49-a120-7ac3d67279cd is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:31,756 - WARNING - Pod uuid 79af9e18-97d8-44e5-a366-cedc14826c58 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:31,763 - WARNING - Pod uuid d8466479-4610-4556-ae22-8c74a428441b is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:36,730 - WARNING - Pod uuid 6b3d91e9-043e-47c6-95e5-984c51727950 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:36,766 - WARNING - Pod uuid 79af9e18-97d8-44e5-a366-cedc14826c58 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:36,766 - WARNING - Pod uuid 95ec114f-0a69-4c49-a120-7ac3d67279cd is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:36,771 - WARNING - Pod uuid d8466479-4610-4556-ae22-8c74a428441b is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:41,747 - WARNING - Pod uuid 6b3d91e9-043e-47c6-95e5-984c51727950 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:41,777 - WARNING - Pod uuid 95ec114f-0a69-4c49-a120-7ac3d67279cd is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:41,778 - WARNING - Pod uuid d8466479-4610-4556-ae22-8c74a428441b is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:41,779 - WARNING - Pod uuid 79af9e18-97d8-44e5-a366-cedc14826c58 is still seen in agent 10.0.0.129 VM list 2026-02-09 23:56:46,753 - DEBUG - Pod 6b3d91e9-043e-47c6-95e5-984c51727950 is not in agent 10.0.0.129 VM list 2026-02-09 23:56:46,758 - DEBUG - VMI 8dfd3c82-0612-11f1-9e0b-fa163e567dec is removed from agent 10.0.0.129 2026-02-09 23:56:46,758 - INFO - Verified that pod ctest-pod-02162332 is removed in agent 2026-02-09 23:56:46,788 - DEBUG - Pod 95ec114f-0a69-4c49-a120-7ac3d67279cd is not in agent 10.0.0.129 VM list 2026-02-09 23:56:46,791 - DEBUG - Pod d8466479-4610-4556-ae22-8c74a428441b is not in agent 10.0.0.129 VM list 2026-02-09 23:56:46,792 - DEBUG - Pod 79af9e18-97d8-44e5-a366-cedc14826c58 is not in agent 10.0.0.129 VM list 2026-02-09 23:56:46,800 - DEBUG - VMI 8e61348a-0612-11f1-9e0b-fa163e567dec is removed from agent 10.0.0.129 2026-02-09 23:56:46,800 - INFO - Verified that pod ctest-pod-17903959 is removed in agent 2026-02-09 23:56:46,801 - DEBUG - VMI 8d35a0e6-0612-11f1-9e0b-fa163e567dec is removed from agent 10.0.0.129 2026-02-09 23:56:46,801 - INFO - Verified that pod ctest-pod-16862472 is removed in agent 2026-02-09 23:56:46,801 - DEBUG - VMI 8d976420-0612-11f1-9e0b-fa163e567dec is removed from agent 10.0.0.129 2026-02-09 23:56:46,801 - INFO - Verified that pod ctest-pod-07128938 is removed in agent 2026-02-09 23:56:46,802 - INFO - Deleting namespace ctest-non-default-93377265 2026-02-09 23:56:46,813 - DEBUG - Namespace ctest-non-default-93377265 still in k8s 2026-02-09 23:56:48,817 - DEBUG - Namespace ctest-non-default-93377265 still in k8s 2026-02-09 23:56:50,821 - DEBUG - Namespace ctest-non-default-93377265 still in k8s 2026-02-09 23:56:52,825 - DEBUG - Namespace ctest-non-default-93377265 is not in k8s 2026-02-09 23:56:52,825 - DEBUG - No need to check project deletion in contrail-api 2026-02-09 23:56:52,825 - INFO - Deleting namespace ctest-new-default-93935537 2026-02-09 23:56:52,846 - DEBUG - Namespace ctest-new-default-93935537 still in k8s 2026-02-09 23:56:54,851 - DEBUG - Namespace ctest-new-default-93935537 still in k8s 2026-02-09 23:56:56,855 - DEBUG - Namespace ctest-new-default-93935537 still in k8s 2026-02-09 23:56:58,858 - DEBUG - Namespace ctest-new-default-93935537 is not in k8s 2026-02-09 23:56:58,858 - DEBUG - No need to check project deletion in contrail-api