2025-01-04 22:15:55,856 - DEBUG - [10.0.0.47]: Running cmd : hostname 2025-01-04 22:15:56,144 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-895-1 2025-01-04 22:15:56,144 - DEBUG - [10.0.0.47]: Running cmd : hostname -f 2025-01-04 22:15:56,189 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-895-1 2025-01-04 22:15:56,189 - DEBUG - [10.0.0.47]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-01-04 22:15:56,280 - DEBUG - Output : NAMES contrail_test_8f3MX9BNM contrail-kubernetes-master_kubemanager_1 analytics_snmp_nodemgr_1 analytics_snmp_topology_1 analytics_snmp_snmp-collector_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_alarm_alarm-gen_1 analytics_api_1 analytics_nodemgr_1 analytics_collector_1 analytics_database_cassandra_1 analytics_database_query-engine_1 analytics_database_nodemgr_1 redis_redis_1 webui_job_1 webui_web_1 control_named_1 control_control_1 control_nodemgr_1 control_dns_1 config_database_nodemgr_1 config_database_zookeeper_1 config_database_cassandra_1 config_database_rabbitmq_1 config_api_devicemgr_1 config_api_api_1 config_api_nodemgr_1 config_api_dnsmasq_1 config_api_svcmonitor_1 config_api_schema_1 2025-01-04 22:15:56,281 - DEBUG - [10.0.0.47]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-01-04 22:15:56,368 - DEBUG - Output : 10.0.0.47/24 172.17.0.1/16 2025-01-04 22:15:56,368 - DEBUG - [10.0.0.47]: Running cmd : getent hosts 10.0.0.47 | head -n 1 | awk '{print $2}' 2025-01-04 22:15:56,412 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-895-1 2025-01-04 22:15:56,514 - DEBUG - [10.0.0.30]: Running cmd : hostname 2025-01-04 22:15:56,746 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-895-1 2025-01-04 22:15:56,747 - DEBUG - [10.0.0.30]: Running cmd : hostname -f 2025-01-04 22:15:56,806 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-895-1 2025-01-04 22:15:56,806 - DEBUG - [10.0.0.30]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-01-04 22:15:56,852 - DEBUG - Output : NAMES vrouter_nodemgr_1 vrouter_vrouter-agent_1 2025-01-04 22:15:56,853 - DEBUG - [10.0.0.30]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-01-04 22:15:56,889 - DEBUG - Output : 10.0.0.30/24 2025-01-04 22:15:56,890 - DEBUG - [10.0.0.30]: Running cmd : getent hosts 10.0.0.30 | head -n 1 | awk '{print $2}' 2025-01-04 22:15:56,958 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-895-1 2025-01-04 22:15:57,058 - INFO - Using existing project ['default-domain', 'k8s-default'](41c25cc0-8c5c-479a-bbac-45d9a638178a) 2025-01-04 22:15:57,173 - INFO - Using existing project ['default-domain', 'k8s-default'](41c25cc0-8c5c-479a-bbac-45d9a638178a) 2025-01-04 22:15:57,316 - INFO - ================================================================================ 2025-01-04 22:15:57,317 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-01-04 22:15:57,317 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-01-04 22:15:57,565 - DEBUG - Requesting: http://10.0.0.30:8085/Snh_AgentXmppConnectionStatusReq? 2025-01-04 22:15:57,577 - DEBUG - Nothing to compare xmpp stats {'10.0.0.30': {'10.0.0.47': '0'}} with 2025-01-04 22:15:57,577 - INFO - Initial checks done. Running the testcase now 2025-01-04 22:15:57,577 - INFO - 2025-01-04 22:15:57,583 - INFO - Namespace default already exists 2025-01-04 22:15:57,584 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-01-04 22:15:57,757 - DEBUG - Output : cluster_project={} 2025-01-04 22:15:57,833 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-01-04 22:15:57,833 - DEBUG - Project uuid in Contrail is 41c25cc0-8c5c-479a-bbac-45d9a638178a 2025-01-04 22:15:57,834 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:15:57,964 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:00,964 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:01,129 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:04,130 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:04,271 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:07,273 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:07,415 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:10,416 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:10,534 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:13,534 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:13,688 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:16,689 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:16,814 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:19,815 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:19,945 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:22,946 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:23,118 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:26,119 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:26,257 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:29,258 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:29,377 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:29,385 - WARNING - Nothing to delete parallely 2025-01-04 22:16:29,625 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Sat Jan 4 22:16:29 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 79 log.info('Initial checks done. Running the testcase now') 80 log.info('') 81 result = function(self, *args, **kwargs) 82 if self.inputs.upgrade: 83 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/k8s_scripts/test_ingress.py in test_ingress_ip_assignment(self=) 39 labels = {'app':app} 40 namespace = self.setup_namespace(name='default') 41 assert namespace.verify_on_setup() 42 43 service = self.setup_http_service(namespace=namespace.name, namespace = namespace.verify_on_setup = > /contrail-test/fixtures/k8s/namespace.py in verify_on_setup(self=) 52 self.name)) 53 return False 54 if not self.verify_namespace_in_kube_manager(): 55 self.logger.error('Namespace %s not seen in Kube Manager' % ( 56 self.name)) self = self.verify_namespace_in_kube_manager = .deco_retry..f_retry of > /contrail-test/tcutils/util.py in f_retry(*args=(,), **kwargs={}) 95 mtries, mdelay = tries, delay # make mutable 96 97 result = f(*args, **kwargs) # first attempt 98 rv = result 99 final = False result undefined f = args = (,) kwargs = {} /contrail-test/fixtures/k8s/namespace.py in verify_namespace_in_kube_manager(self=) 120 @retry(delay=2, tries=10) 121 def verify_namespace_in_kube_manager(self): 122 km_h = self.connections.get_kube_manager_h() 123 self.namespace_info = km_h.get_namespace_info(ns_uuid = self.uuid) 124 if self.namespace_info: km_h undefined self = self.connections = self.connections.get_kube_manager_h = > /contrail-test/common/connections.py in get_kube_manager_h(self=, refresh=False) 349 self._get_kube_manager_h(refresh=refresh) 350 msg = "Kubernetes manager service is not up" 351 assert self._kube_manager_inspect is not None, msg 352 return self._kube_manager_inspect 353 self = self._kube_manager_inspect = None msg = 'Kubernetes manager service is not up' AssertionError: Kubernetes manager service is not up __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('Kubernetes manager service is not up',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 81, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/k8s_scripts/test_ingress.py", line 41, in test_ingress_ip_assignment assert namespace.verify_on_setup() File "/contrail-test/fixtures/k8s/namespace.py", line 54, in verify_on_setup if not self.verify_namespace_in_kube_manager(): File "/contrail-test/tcutils/util.py", line 97, in f_retry result = f(*args, **kwargs) # first attempt File "/contrail-test/fixtures/k8s/namespace.py", line 122, in verify_namespace_in_kube_manager km_h = self.connections.get_kube_manager_h() File "/contrail-test/common/connections.py", line 351, in get_kube_manager_h assert self._kube_manager_inspect is not None, msg AssertionError: Kubernetes manager service is not up 2025-01-04 22:16:29,625 - DEBUG - Requesting: http://10.0.0.30:8085/Snh_AgentXmppConnectionStatusReq? 2025-01-04 22:16:29,633 - DEBUG - No XMPP flaps were noticed during the test 2025-01-04 22:16:29,633 - INFO -
2025-01-04 22:16:29,633 - INFO - END TEST : test_ingress_ip_assignment : FAILED[0:00:32]
2025-01-04 22:16:29,633 - INFO - -------------------------------------------------------------------------------- 2025-01-04 22:16:36,157 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-01-04 22:16:36,279 - DEBUG - Output : cluster_project={} 2025-01-04 22:16:36,345 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:36,487 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:39,488 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:39,617 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:42,618 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:42,773 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:45,774 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:45,925 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:48,926 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:49,098 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:52,099 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:52,248 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:55,249 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:55,366 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:16:58,367 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:16:58,519 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:17:01,520 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:17:01,651 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:17:04,652 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:17:04,792 - DEBUG - Output : /bin/bash: netstat: command not found 2025-01-04 22:17:07,793 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-01-04 22:17:07,943 - DEBUG - Output : /bin/bash: netstat: command not found