2025-01-05 13:37:13,678 - DEBUG - [10.0.0.44]: Running cmd : hostname 2025-01-05 13:37:13,951 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-896-1 2025-01-05 13:37:13,951 - DEBUG - [10.0.0.44]: Running cmd : hostname -f 2025-01-05 13:37:13,985 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-896-1 2025-01-05 13:37:13,985 - DEBUG - [10.0.0.44]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-01-05 13:37:14,061 - DEBUG - Output : NAMES contrail_test_KSrOUsnLs contrail-kubernetes-master_kubemanager_1 webui_job_1 webui_web_1 analytics_snmp_nodemgr_1 analytics_snmp_snmp-collector_1 analytics_snmp_topology_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_collector_1 analytics_api_1 analytics_database_cassandra_1 analytics_database_nodemgr_1 analytics_database_query-engine_1 redis_redis_1 control_named_1 control_dns_1 control_nodemgr_1 control_control_1 config_database_zookeeper_1 config_database_cassandra_1 config_database_rabbitmq_1 config_database_nodemgr_1 config_api_devicemgr_1 config_api_nodemgr_1 config_api_api_1 config_api_dnsmasq_1 config_api_svcmonitor_1 config_api_schema_1 2025-01-05 13:37:14,061 - DEBUG - [10.0.0.44]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-01-05 13:37:14,138 - DEBUG - Output : 10.0.0.44/24 172.17.0.1/16 2025-01-05 13:37:14,138 - DEBUG - [10.0.0.44]: Running cmd : getent hosts 10.0.0.44 | head -n 1 | awk '{print $2}' 2025-01-05 13:37:14,171 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-896-1 2025-01-05 13:37:14,245 - DEBUG - [10.0.0.28]: Running cmd : hostname 2025-01-05 13:37:14,467 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-896-1 2025-01-05 13:37:14,468 - DEBUG - [10.0.0.28]: Running cmd : hostname -f 2025-01-05 13:37:14,503 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-896-1 2025-01-05 13:37:14,504 - DEBUG - [10.0.0.28]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-01-05 13:37:14,559 - DEBUG - Output : NAMES vrouter_nodemgr_1 vrouter_vrouter-agent_1 2025-01-05 13:37:14,560 - DEBUG - [10.0.0.28]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-01-05 13:37:14,627 - DEBUG - Output : 10.0.0.28/24 2025-01-05 13:37:14,628 - DEBUG - [10.0.0.28]: Running cmd : getent hosts 10.0.0.28 | head -n 1 | awk '{print $2}' 2025-01-05 13:37:14,662 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-896-1 2025-01-05 13:37:14,749 - INFO - Using existing project ['default-domain', 'k8s-default'](cffea915-5e30-4880-94d9-b3f046c76d66) 2025-01-05 13:37:14,811 - INFO - Using existing project ['default-domain', 'k8s-default'](cffea915-5e30-4880-94d9-b3f046c76d66) 2025-01-05 13:37:14,968 - INFO - ================================================================================ 2025-01-05 13:37:14,968 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-01-05 13:37:14,969 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-01-05 13:37:15,174 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_AgentXmppConnectionStatusReq? 2025-01-05 13:37:15,182 - DEBUG - Nothing to compare xmpp stats {'10.0.0.28': {'10.0.0.44': '0'}} with 2025-01-05 13:37:15,182 - INFO - Initial checks done. Running the testcase now 2025-01-05 13:37:15,182 - INFO - 2025-01-05 13:37:15,186 - INFO - Namespace default already exists 2025-01-05 13:37:15,186 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-01-05 13:37:15,302 - DEBUG - Output : cluster_project={} 2025-01-05 13:37:15,393 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-01-05 13:37:15,394 - DEBUG - Project uuid in Contrail is cffea915-5e30-4880-94d9-b3f046c76d66 2025-01-05 13:37:15,394 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:15,500 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:18,500 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:18,611 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:21,612 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:21,726 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:24,727 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:24,844 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:27,845 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:27,954 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:30,955 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:31,062 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:34,063 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:34,158 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:37,160 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:37,290 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:40,291 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:40,429 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:43,430 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:43,537 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:46,538 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:46,666 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:46,677 - WARNING - Nothing to delete parallely 2025-01-05 13:37:46,936 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Sun Jan 5 13:37:46 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 79 log.info('Initial checks done. Running the testcase now') 80 log.info('') 81 result = function(self, *args, **kwargs) 82 if self.inputs.upgrade: 83 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/k8s_scripts/test_ingress.py in test_ingress_ip_assignment(self=) 39 labels = {'app':app} 40 namespace = self.setup_namespace(name='default') 41 assert namespace.verify_on_setup() 42 43 service = self.setup_http_service(namespace=namespace.name, namespace = namespace.verify_on_setup = > /contrail-test/fixtures/k8s/namespace.py in verify_on_setup(self=) 52 self.name)) 53 return False 54 if not self.verify_namespace_in_kube_manager(): 55 self.logger.error('Namespace %s not seen in Kube Manager' % ( 56 self.name)) self = self.verify_namespace_in_kube_manager = .deco_retry..f_retry of > /contrail-test/tcutils/util.py in f_retry(*args=(,), **kwargs={}) 95 mtries, mdelay = tries, delay # make mutable 96 97 result = f(*args, **kwargs) # first attempt 98 rv = result 99 final = False result undefined f = args = (,) kwargs = {} /contrail-test/fixtures/k8s/namespace.py in verify_namespace_in_kube_manager(self=) 120 @retry(delay=2, tries=10) 121 def verify_namespace_in_kube_manager(self): 122 km_h = self.connections.get_kube_manager_h() 123 self.namespace_info = km_h.get_namespace_info(ns_uuid = self.uuid) 124 if self.namespace_info: km_h undefined self = self.connections = self.connections.get_kube_manager_h = > /contrail-test/common/connections.py in get_kube_manager_h(self=, refresh=False) 349 self._get_kube_manager_h(refresh=refresh) 350 msg = "Kubernetes manager service is not up" 351 assert self._kube_manager_inspect is not None, msg 352 return self._kube_manager_inspect 353 self = self._kube_manager_inspect = None msg = 'Kubernetes manager service is not up' AssertionError: Kubernetes manager service is not up __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('Kubernetes manager service is not up',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 81, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/k8s_scripts/test_ingress.py", line 41, in test_ingress_ip_assignment assert namespace.verify_on_setup() File "/contrail-test/fixtures/k8s/namespace.py", line 54, in verify_on_setup if not self.verify_namespace_in_kube_manager(): File "/contrail-test/tcutils/util.py", line 97, in f_retry result = f(*args, **kwargs) # first attempt File "/contrail-test/fixtures/k8s/namespace.py", line 122, in verify_namespace_in_kube_manager km_h = self.connections.get_kube_manager_h() File "/contrail-test/common/connections.py", line 351, in get_kube_manager_h assert self._kube_manager_inspect is not None, msg AssertionError: Kubernetes manager service is not up 2025-01-05 13:37:46,937 - DEBUG - Requesting: http://10.0.0.28:8085/Snh_AgentXmppConnectionStatusReq? 2025-01-05 13:37:46,943 - DEBUG - No XMPP flaps were noticed during the test 2025-01-05 13:37:46,944 - INFO -
2025-01-05 13:37:46,944 - INFO - END TEST : test_ingress_ip_assignment : FAILED[0:00:32]
2025-01-05 13:37:46,944 - INFO - -------------------------------------------------------------------------------- 2025-01-05 13:37:53,338 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-01-05 13:37:53,483 - DEBUG - Output : cluster_project={} 2025-01-05 13:37:53,536 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:53,662 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:56,663 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:37:56,789 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:37:59,790 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:38:00,098 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:38:03,099 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:38:03,219 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:38:06,220 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:38:06,336 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:38:09,337 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:38:09,492 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:38:12,493 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:38:12,641 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:38:15,642 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:38:15,750 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:38:18,751 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:38:18,889 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:38:21,890 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:38:22,019 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-05 13:38:25,020 - DEBUG - [10.0.0.44]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-05 13:38:25,137 - DEBUG - Output : /bin/bash: netstat: command not found