2025-03-27 16:31:49,443 - DEBUG - [10.0.0.52]: Running cmd : hostname 2025-03-27 16:31:49,857 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-292-1.novalocal 2025-03-27 16:31:49,857 - DEBUG - [10.0.0.52]: Running cmd : hostname -f 2025-03-27 16:31:50,020 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-292-1.novalocal 2025-03-27 16:31:50,020 - DEBUG - [10.0.0.52]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-03-27 16:31:50,270 - DEBUG - Output : NAMES contrail_test_xrwWjIQGC vrouter_vrouter-agent_1 vrouter_nodemgr_1 vrouter_node-init_1 rsyslogd_rsyslogd_1 kubemanager_k8s_kubemanager_1 kubemanager_k8s_node-init_1 cni_vrouter-cni_1 analytics_snmp_topology_1 analytics_snmp_nodemgr_1 analytics_snmp_snmp-collector_1 analytics_snmp_node-init_1 analytics_alarm_nodemgr_1 analytics_alarm_alarm-gen_1 analytics_alarm_node-init_1 analytics_alarm_kafka_1 analytics_api_1 analytics_collector_1 analytics_nodemgr_1 analytics_node-init_1 analytics_database_nodemgr_1 analytics_database_query-engine_1 analytics_database_cassandra_1 analytics_database_node-init_1 control_named_1 control_dns_1 control_nodemgr_1 control_control_1 control_node-init_1 webui_web_1 webui_job_1 webui_node-init_1 config_devicemgr_1 config_svcmonitor_1 config_api_1 config_schema_1 config_dnsmasq_1 config_nodemgr_1 config_node-init_1 config_database_nodemgr_1 config_database_rabbitmq_1 config_database_cassandra_1 config_database_zookeeper_1 config_database_node-init_1 redis_redis_1 2025-03-27 16:31:50,271 - DEBUG - [10.0.0.52]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-03-27 16:31:50,447 - DEBUG - Output : 172.17.0.1/16 10.0.0.52/24 2025-03-27 16:31:50,448 - DEBUG - [10.0.0.52]: Running cmd : getent hosts 10.0.0.52 | head -n 1 | awk '{print $2}' 2025-03-27 16:31:50,626 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-292-1.novalocal 2025-03-27 16:31:50,626 - DEBUG - [10.0.0.52]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-03-27 16:31:50,787 - DEBUG - Output : 10.0.0.52/24 2025-03-27 16:31:51,156 - INFO - Using existing project ['default-domain', 'k8s-default'](8b09e356-526b-4243-a516-4030494f54ef) 2025-03-27 16:31:51,273 - INFO - Using existing project ['default-domain', 'k8s-default'](8b09e356-526b-4243-a516-4030494f54ef) 2025-03-27 16:31:51,466 - INFO - ================================================================================ 2025-03-27 16:31:51,467 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-03-27 16:31:51,468 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-03-27 16:31:51,755 - DEBUG - Requesting: http://10.0.0.52:8085/Snh_AgentXmppConnectionStatusReq? 2025-03-27 16:31:51,769 - DEBUG - Nothing to compare xmpp stats {'10.0.0.52': {'10.0.0.52': '0'}} with 2025-03-27 16:31:51,769 - INFO - Initial checks done. Running the testcase now 2025-03-27 16:31:51,769 - INFO - 2025-03-27 16:31:51,775 - INFO - Namespace default already exists 2025-03-27 16:31:51,776 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-03-27 16:31:51,952 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:31:52,034 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-03-27 16:31:52,034 - DEBUG - Project uuid in Contrail is 8b09e356-526b-4243-a516-4030494f54ef 2025-03-27 16:31:52,034 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:31:52,207 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:31:55,209 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:31:55,400 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:31:58,402 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:31:58,552 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:01,554 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:01,716 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:04,717 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:04,880 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:07,880 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:08,044 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:11,046 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:11,219 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:14,219 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:14,370 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:17,372 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:17,538 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:20,539 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:20,705 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:23,706 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:23,868 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:23,879 - WARNING - Nothing to delete parallely 2025-03-27 16:32:24,214 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Thu Mar 27 16:32:23 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/k8s_scripts/test_ingress.py in test_ingress_ip_assignment(self=) 39 labels = {'app':app} 40 namespace = self.setup_namespace(name='default') 41 assert namespace.verify_on_setup() 42 43 service = self.setup_http_service(namespace=namespace.name, namespace = namespace.verify_on_setup = > /contrail-test/fixtures/k8s/namespace.py in verify_on_setup(self=) 51 self.name)) 52 return False 53 if not self.verify_namespace_in_kube_manager(): 54 self.logger.error('Namespace %s not seen in Kube Manager' % ( 55 self.name)) self = self.verify_namespace_in_kube_manager = .deco_retry..f_retry of > /contrail-test/tcutils/util.py in f_retry(*args=(,), **kwargs={}) 82 mtries, mdelay = tries, delay # make mutable 83 84 result = f(*args, **kwargs) # first attempt 85 rv = result 86 final = False result undefined f = args = (,) kwargs = {} /contrail-test/fixtures/k8s/namespace.py in verify_namespace_in_kube_manager(self=) 119 @retry(delay=2, tries=10) 120 def verify_namespace_in_kube_manager(self): 121 km_h = self.connections.get_kube_manager_h() 122 self.namespace_info = km_h.get_namespace_info(ns_uuid = self.uuid) 123 if self.namespace_info: km_h undefined self = self.connections = self.connections.get_kube_manager_h = > /contrail-test/common/connections.py in get_kube_manager_h(self=, refresh=False) 306 self._get_kube_manager_h(refresh=refresh) 307 msg = "Kubernetes manager service is not up" 308 assert self._kube_manager_inspect is not None, msg 309 return self._kube_manager_inspect 310 self = self._kube_manager_inspect = None msg = 'Kubernetes manager service is not up' AssertionError: Kubernetes manager service is not up __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('Kubernetes manager service is not up',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/k8s_scripts/test_ingress.py", line 41, in test_ingress_ip_assignment assert namespace.verify_on_setup() File "/contrail-test/fixtures/k8s/namespace.py", line 53, in verify_on_setup if not self.verify_namespace_in_kube_manager(): File "/contrail-test/tcutils/util.py", line 84, in f_retry result = f(*args, **kwargs) # first attempt File "/contrail-test/fixtures/k8s/namespace.py", line 121, in verify_namespace_in_kube_manager km_h = self.connections.get_kube_manager_h() File "/contrail-test/common/connections.py", line 308, in get_kube_manager_h assert self._kube_manager_inspect is not None, msg AssertionError: Kubernetes manager service is not up 2025-03-27 16:32:24,214 - DEBUG - Requesting: http://10.0.0.52:8085/Snh_AgentXmppConnectionStatusReq? 2025-03-27 16:32:24,224 - DEBUG - No XMPP flaps were noticed during the test 2025-03-27 16:32:24,224 - INFO -
2025-03-27 16:32:24,224 - INFO - END TEST : test_ingress_ip_assignment : FAILED[0:00:33]
2025-03-27 16:32:24,224 - INFO - -------------------------------------------------------------------------------- 2025-03-27 16:32:30,761 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-03-27 16:32:30,972 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:31,087 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:31,254 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:34,256 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:34,409 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:37,410 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:37,563 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:40,564 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:40,738 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:43,739 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:43,891 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:46,892 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:47,047 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:50,048 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:50,248 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:53,249 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:53,394 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:56,395 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:56,579 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:32:59,581 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:32:59,758 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running 2025-03-27 16:33:02,759 - DEBUG - [10.0.0.52]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 16:33:02,937 - DEBUG - Output : Error response from daemon: container d0fa19255b008220372491f1ec7bf74c4f0810d048f7f515b57a2a5cb4330e10 is not running