2025-03-28 11:18:42,654 - DEBUG - [10.0.0.27]: Running cmd : hostname 2025-03-28 11:18:42,944 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-294-1.novalocal 2025-03-28 11:18:42,944 - DEBUG - [10.0.0.27]: Running cmd : hostname -f 2025-03-28 11:18:43,064 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-294-1.novalocal 2025-03-28 11:18:43,065 - DEBUG - [10.0.0.27]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-03-28 11:18:43,257 - DEBUG - Output : NAMES contrail_test_oajjczdqU vrouter_vrouter-agent_1 vrouter_nodemgr_1 vrouter_node-init_1 rsyslogd_rsyslogd_1 kubemanager_k8s_kubemanager_1 kubemanager_k8s_node-init_1 cni_vrouter-cni_1 analytics_snmp_topology_1 analytics_snmp_nodemgr_1 analytics_snmp_snmp-collector_1 analytics_snmp_node-init_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_node-init_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_api_1 analytics_collector_1 analytics_node-init_1 analytics_database_nodemgr_1 analytics_database_query-engine_1 analytics_database_cassandra_1 analytics_database_node-init_1 control_named_1 control_control_1 control_nodemgr_1 control_dns_1 control_node-init_1 webui_job_1 webui_web_1 webui_node-init_1 config_devicemgr_1 config_dnsmasq_1 config_api_1 config_nodemgr_1 config_svcmonitor_1 config_schema_1 config_node-init_1 config_database_nodemgr_1 config_database_rabbitmq_1 config_database_node-init_1 config_database_zookeeper_1 config_database_cassandra_1 redis_redis_1 2025-03-28 11:18:43,257 - DEBUG - [10.0.0.27]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-03-28 11:18:43,391 - DEBUG - Output : 172.17.0.1/16 10.0.0.27/24 2025-03-28 11:18:43,392 - DEBUG - [10.0.0.27]: Running cmd : getent hosts 10.0.0.27 | head -n 1 | awk '{print $2}' 2025-03-28 11:18:43,516 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-294-1.novalocal 2025-03-28 11:18:43,516 - DEBUG - [10.0.0.27]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-03-28 11:18:43,653 - DEBUG - Output : 10.0.0.27/24 2025-03-28 11:18:43,928 - INFO - Using existing project ['default-domain', 'k8s-default'](2e3530a3-d30d-41b6-8caf-745213029c67) 2025-03-28 11:18:44,009 - INFO - Using existing project ['default-domain', 'k8s-default'](2e3530a3-d30d-41b6-8caf-745213029c67) 2025-03-28 11:18:44,155 - INFO - ================================================================================ 2025-03-28 11:18:44,155 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-03-28 11:18:44,156 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-03-28 11:18:44,388 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_AgentXmppConnectionStatusReq? 2025-03-28 11:18:44,395 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.0.0.27': '0'}} with 2025-03-28 11:18:44,395 - INFO - Initial checks done. Running the testcase now 2025-03-28 11:18:44,395 - INFO - 2025-03-28 11:18:44,399 - INFO - Namespace default already exists 2025-03-28 11:18:44,400 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-03-28 11:18:44,527 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:18:44,579 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-03-28 11:18:44,579 - DEBUG - Project uuid in Contrail is 2e3530a3-d30d-41b6-8caf-745213029c67 2025-03-28 11:18:44,579 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:18:44,688 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:18:47,689 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:18:47,806 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:18:50,807 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:18:50,924 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:18:53,925 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:18:54,031 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:18:57,031 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:18:57,127 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:00,128 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:00,224 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:03,225 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:03,321 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:06,322 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:06,428 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:09,428 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:09,545 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:12,547 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:12,684 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:15,685 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:15,792 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:15,801 - WARNING - Nothing to delete parallely 2025-03-28 11:19:16,062 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Fri Mar 28 11:19:15 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/k8s_scripts/test_ingress.py in test_ingress_ip_assignment(self=) 39 labels = {'app':app} 40 namespace = self.setup_namespace(name='default') 41 assert namespace.verify_on_setup() 42 43 service = self.setup_http_service(namespace=namespace.name, namespace = namespace.verify_on_setup = > /contrail-test/fixtures/k8s/namespace.py in verify_on_setup(self=) 51 self.name)) 52 return False 53 if not self.verify_namespace_in_kube_manager(): 54 self.logger.error('Namespace %s not seen in Kube Manager' % ( 55 self.name)) self = self.verify_namespace_in_kube_manager = .deco_retry..f_retry of > /contrail-test/tcutils/util.py in f_retry(*args=(,), **kwargs={}) 82 mtries, mdelay = tries, delay # make mutable 83 84 result = f(*args, **kwargs) # first attempt 85 rv = result 86 final = False result undefined f = args = (,) kwargs = {} /contrail-test/fixtures/k8s/namespace.py in verify_namespace_in_kube_manager(self=) 119 @retry(delay=2, tries=10) 120 def verify_namespace_in_kube_manager(self): 121 km_h = self.connections.get_kube_manager_h() 122 self.namespace_info = km_h.get_namespace_info(ns_uuid = self.uuid) 123 if self.namespace_info: km_h undefined self = self.connections = self.connections.get_kube_manager_h = > /contrail-test/common/connections.py in get_kube_manager_h(self=, refresh=False) 306 self._get_kube_manager_h(refresh=refresh) 307 msg = "Kubernetes manager service is not up" 308 assert self._kube_manager_inspect is not None, msg 309 return self._kube_manager_inspect 310 self = self._kube_manager_inspect = None msg = 'Kubernetes manager service is not up' AssertionError: Kubernetes manager service is not up __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('Kubernetes manager service is not up',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/k8s_scripts/test_ingress.py", line 41, in test_ingress_ip_assignment assert namespace.verify_on_setup() File "/contrail-test/fixtures/k8s/namespace.py", line 53, in verify_on_setup if not self.verify_namespace_in_kube_manager(): File "/contrail-test/tcutils/util.py", line 84, in f_retry result = f(*args, **kwargs) # first attempt File "/contrail-test/fixtures/k8s/namespace.py", line 121, in verify_namespace_in_kube_manager km_h = self.connections.get_kube_manager_h() File "/contrail-test/common/connections.py", line 308, in get_kube_manager_h assert self._kube_manager_inspect is not None, msg AssertionError: Kubernetes manager service is not up 2025-03-28 11:19:16,063 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_AgentXmppConnectionStatusReq? 2025-03-28 11:19:16,070 - DEBUG - No XMPP flaps were noticed during the test 2025-03-28 11:19:16,070 - INFO -
2025-03-28 11:19:16,070 - INFO - END TEST : test_ingress_ip_assignment : FAILED[0:00:32]
2025-03-28 11:19:16,070 - INFO - -------------------------------------------------------------------------------- 2025-03-28 11:19:22,430 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-03-28 11:19:22,537 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:22,582 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:22,712 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:25,714 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:25,824 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:28,825 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:28,932 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:31,933 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:32,039 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:35,041 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:35,159 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:38,160 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:38,290 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:41,291 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:41,449 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:44,451 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:44,558 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:47,559 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:47,687 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:50,688 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:50,797 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running 2025-03-28 11:19:53,798 - DEBUG - [10.0.0.27]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-28 11:19:53,922 - DEBUG - Output : Error response from daemon: container a634d9ba41ef4d6fcb7a0631d5ec30ec46b747d1e8383f98d9c7b2f70db7c2b1 is not running