2026-08-31 02:25:41,074 - INFO - Using existing project ['default-domain', 'admin'](d21f6d7f-753d-4756-9c08-92953335e433) 2026-08-31 02:25:41,082 - INFO - ================================================================================ 2026-08-31 02:25:41,082 - INFO - STARTING TEST : test_contrail_status 2026-08-31 02:25:41,082 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2026-08-31 02:25:41,779 - DEBUG - Nothing to compare xmpp stats {'10.0.0.186': {'10.20.0.14': '0', '10.20.0.254': '0'}, '10.0.0.176': {'10.20.0.14': '0', '10.20.0.25': '0'}} with 2026-08-31 02:25:41,779 - INFO - Initial checks done. Running the testcase now 2026-08-31 02:25:41,779 - INFO - 2026-08-31 02:25:41,847 - INFO - 10.0.0.50 2026-08-31 02:25:41,847 - INFO - config 2026-08-31 02:25:41,853 - INFO - config-nodemgr:active 2026-08-31 02:25:41,857 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down) 2026-08-31 02:25:41,862 - INFO - schema:active 2026-08-31 02:25:41,866 - INFO - svc-monitor:active 2026-08-31 02:25:41,870 - INFO - device-manager:active 2026-08-31 02:25:41,870 - INFO - config-database 2026-08-31 02:25:41,870 - INFO - config-cassandra:active 2026-08-31 02:25:41,870 - INFO - config-zookeeper:active 2026-08-31 02:25:41,870 - INFO - config-rabbitmq:active 2026-08-31 02:25:41,870 - INFO - control 2026-08-31 02:25:41,875 - INFO - control-nodemgr:active 2026-08-31 02:25:41,880 - INFO - control:active 2026-08-31 02:25:41,880 - INFO - named:active 2026-08-31 02:25:41,886 - INFO - dns:active 2026-08-31 02:25:41,886 - INFO - analytics 2026-08-31 02:25:41,890 - INFO - analytics-nodemgr:active 2026-08-31 02:25:41,895 - INFO - analytics-api:active 2026-08-31 02:25:41,899 - INFO - collector:active 2026-08-31 02:25:41,900 - INFO - analytics-database 2026-08-31 02:25:41,900 - INFO - analytics-cassandra:active 2026-08-31 02:25:41,904 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:25:41,911 - INFO - query-engine:active 2026-08-31 02:25:41,911 - INFO - webui 2026-08-31 02:25:41,911 - INFO - webui:active 2026-08-31 02:25:41,911 - INFO - webui-middleware:active 2026-08-31 02:25:41,911 - INFO - redis:active 2026-08-31 02:25:41,911 - INFO - analytics_snmp 2026-08-31 02:25:41,916 - INFO - snmp-collector:active 2026-08-31 02:25:41,917 - INFO - snmp-topology:active 2026-08-31 02:25:41,917 - INFO - analytics_alarm 2026-08-31 02:25:41,917 - INFO - alarmgen:active 2026-08-31 02:25:41,972 - INFO - 10.0.0.185 2026-08-31 02:25:41,973 - INFO - config 2026-08-31 02:25:41,979 - INFO - config-nodemgr:active 2026-08-31 02:25:41,984 - INFO - api-server:active 2026-08-31 02:25:41,987 - INFO - schema:backup 2026-08-31 02:25:41,990 - INFO - svc-monitor:backup 2026-08-31 02:25:41,993 - INFO - device-manager:backup 2026-08-31 02:25:41,993 - INFO - config-database 2026-08-31 02:25:41,993 - INFO - config-cassandra:active 2026-08-31 02:25:41,993 - INFO - config-zookeeper:active 2026-08-31 02:25:41,993 - INFO - config-rabbitmq:active 2026-08-31 02:25:41,993 - INFO - control 2026-08-31 02:25:41,998 - INFO - control-nodemgr:active 2026-08-31 02:25:42,003 - INFO - control:active 2026-08-31 02:25:42,003 - INFO - named:active 2026-08-31 02:25:42,009 - INFO - dns:active 2026-08-31 02:25:42,009 - INFO - analytics 2026-08-31 02:25:42,015 - INFO - analytics-nodemgr:active 2026-08-31 02:25:42,020 - INFO - analytics-api:active 2026-08-31 02:25:42,026 - INFO - collector:active 2026-08-31 02:25:42,026 - INFO - analytics-database 2026-08-31 02:25:42,026 - INFO - analytics-cassandra:active 2026-08-31 02:25:42,032 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:25:42,039 - INFO - query-engine:active 2026-08-31 02:25:42,039 - INFO - webui 2026-08-31 02:25:42,039 - INFO - webui:active 2026-08-31 02:25:42,039 - INFO - webui-middleware:active 2026-08-31 02:25:42,039 - INFO - redis:active 2026-08-31 02:25:42,039 - INFO - analytics_snmp 2026-08-31 02:25:42,045 - INFO - snmp-collector:active 2026-08-31 02:25:42,045 - INFO - snmp-topology:active 2026-08-31 02:25:42,045 - INFO - analytics_alarm 2026-08-31 02:25:42,045 - INFO - alarmgen:active 2026-08-31 02:25:42,102 - INFO - 10.0.0.38 2026-08-31 02:25:42,102 - INFO - config 2026-08-31 02:25:42,108 - INFO - config-nodemgr:active 2026-08-31 02:25:42,113 - INFO - api-server:active 2026-08-31 02:25:42,116 - INFO - schema:backup 2026-08-31 02:25:42,119 - INFO - svc-monitor:backup 2026-08-31 02:25:42,123 - INFO - device-manager:backup 2026-08-31 02:25:42,123 - INFO - config-database 2026-08-31 02:25:42,123 - INFO - config-cassandra:active 2026-08-31 02:25:42,123 - INFO - config-zookeeper:active 2026-08-31 02:25:42,123 - INFO - config-rabbitmq:active 2026-08-31 02:25:42,123 - INFO - control 2026-08-31 02:25:42,128 - INFO - control-nodemgr:active 2026-08-31 02:25:42,134 - INFO - control:active 2026-08-31 02:25:42,134 - INFO - named:active 2026-08-31 02:25:42,138 - INFO - dns:active 2026-08-31 02:25:42,139 - INFO - analytics 2026-08-31 02:25:42,143 - INFO - analytics-nodemgr:active 2026-08-31 02:25:42,148 - INFO - analytics-api:active 2026-08-31 02:25:42,153 - INFO - collector:active 2026-08-31 02:25:42,153 - INFO - analytics-database 2026-08-31 02:25:42,153 - INFO - analytics-cassandra:active 2026-08-31 02:25:42,158 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:25:42,166 - INFO - query-engine:active 2026-08-31 02:25:42,166 - INFO - webui 2026-08-31 02:25:42,166 - INFO - webui:active 2026-08-31 02:25:42,166 - INFO - webui-middleware:active 2026-08-31 02:25:42,166 - INFO - redis:active 2026-08-31 02:25:42,166 - INFO - analytics_snmp 2026-08-31 02:25:42,171 - INFO - snmp-collector:active 2026-08-31 02:25:42,171 - INFO - snmp-topology:active 2026-08-31 02:25:42,171 - INFO - analytics_alarm 2026-08-31 02:25:42,171 - INFO - alarmgen:active 2026-08-31 02:25:42,217 - INFO - 10.0.0.186 2026-08-31 02:25:42,217 - INFO - vrouter 2026-08-31 02:25:42,223 - INFO - vrouter-nodemgr:active 2026-08-31 02:25:42,228 - INFO - agent:active 2026-08-31 02:25:42,284 - INFO - 10.0.0.176 2026-08-31 02:25:42,284 - INFO - vrouter 2026-08-31 02:25:42,290 - INFO - vrouter-nodemgr:active 2026-08-31 02:25:42,296 - INFO - agent:active 2026-08-31 02:25:42,297 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down'}}}) 2026-08-31 02:25:42,297 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 0 2026-08-31 02:25:52,356 - INFO - 10.0.0.50 2026-08-31 02:25:52,356 - INFO - config 2026-08-31 02:25:52,361 - INFO - config-nodemgr:active 2026-08-31 02:25:52,365 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down) 2026-08-31 02:25:52,368 - INFO - schema:active 2026-08-31 02:25:52,372 - INFO - svc-monitor:active 2026-08-31 02:25:52,375 - INFO - device-manager:active 2026-08-31 02:25:52,376 - INFO - config-database 2026-08-31 02:25:52,376 - INFO - config-cassandra:active 2026-08-31 02:25:52,376 - INFO - config-zookeeper:active 2026-08-31 02:25:52,376 - INFO - config-rabbitmq:active 2026-08-31 02:25:52,376 - INFO - control 2026-08-31 02:25:52,380 - INFO - control-nodemgr:active 2026-08-31 02:25:52,387 - INFO - control:active 2026-08-31 02:25:52,388 - INFO - named:active 2026-08-31 02:25:52,397 - INFO - dns:active 2026-08-31 02:25:52,398 - INFO - analytics 2026-08-31 02:25:52,402 - INFO - analytics-nodemgr:active 2026-08-31 02:25:52,405 - INFO - analytics-api:active 2026-08-31 02:25:52,410 - INFO - collector:active 2026-08-31 02:25:52,410 - INFO - analytics-database 2026-08-31 02:25:52,410 - INFO - analytics-cassandra:active 2026-08-31 02:25:52,415 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:25:52,422 - INFO - query-engine:active 2026-08-31 02:25:52,423 - INFO - webui 2026-08-31 02:25:52,423 - INFO - webui:active 2026-08-31 02:25:52,423 - INFO - webui-middleware:active 2026-08-31 02:25:52,423 - INFO - redis:active 2026-08-31 02:25:52,423 - INFO - analytics_snmp 2026-08-31 02:25:52,426 - INFO - snmp-collector:active 2026-08-31 02:25:52,427 - INFO - snmp-topology:active 2026-08-31 02:25:52,427 - INFO - analytics_alarm 2026-08-31 02:25:52,427 - INFO - alarmgen:active 2026-08-31 02:25:52,494 - INFO - 10.0.0.185 2026-08-31 02:25:52,494 - INFO - config 2026-08-31 02:25:52,499 - INFO - config-nodemgr:active 2026-08-31 02:25:52,504 - INFO - api-server:active 2026-08-31 02:25:52,507 - INFO - schema:backup 2026-08-31 02:25:52,510 - INFO - svc-monitor:backup 2026-08-31 02:25:52,513 - INFO - device-manager:backup 2026-08-31 02:25:52,513 - INFO - config-database 2026-08-31 02:25:52,513 - INFO - config-cassandra:active 2026-08-31 02:25:52,513 - INFO - config-zookeeper:active 2026-08-31 02:25:52,513 - INFO - config-rabbitmq:active 2026-08-31 02:25:52,513 - INFO - control 2026-08-31 02:25:52,517 - INFO - control-nodemgr:active 2026-08-31 02:25:52,522 - INFO - control:active 2026-08-31 02:25:52,522 - INFO - named:active 2026-08-31 02:25:52,526 - INFO - dns:active 2026-08-31 02:25:52,526 - INFO - analytics 2026-08-31 02:25:52,532 - INFO - analytics-nodemgr:active 2026-08-31 02:25:52,536 - INFO - analytics-api:active 2026-08-31 02:25:52,541 - INFO - collector:active 2026-08-31 02:25:52,541 - INFO - analytics-database 2026-08-31 02:25:52,541 - INFO - analytics-cassandra:active 2026-08-31 02:25:52,546 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:25:52,552 - INFO - query-engine:active 2026-08-31 02:25:52,552 - INFO - webui 2026-08-31 02:25:52,552 - INFO - webui:active 2026-08-31 02:25:52,552 - INFO - webui-middleware:active 2026-08-31 02:25:52,552 - INFO - redis:active 2026-08-31 02:25:52,552 - INFO - analytics_snmp 2026-08-31 02:25:52,556 - INFO - snmp-collector:active 2026-08-31 02:25:52,556 - INFO - snmp-topology:active 2026-08-31 02:25:52,556 - INFO - analytics_alarm 2026-08-31 02:25:52,556 - INFO - alarmgen:active 2026-08-31 02:25:52,611 - INFO - 10.0.0.38 2026-08-31 02:25:52,611 - INFO - config 2026-08-31 02:25:52,617 - INFO - config-nodemgr:active 2026-08-31 02:25:52,622 - INFO - api-server:active 2026-08-31 02:25:52,626 - INFO - schema:backup 2026-08-31 02:25:52,628 - INFO - svc-monitor:backup 2026-08-31 02:25:52,632 - INFO - device-manager:backup 2026-08-31 02:25:52,632 - INFO - config-database 2026-08-31 02:25:52,632 - INFO - config-cassandra:active 2026-08-31 02:25:52,632 - INFO - config-zookeeper:active 2026-08-31 02:25:52,632 - INFO - config-rabbitmq:active 2026-08-31 02:25:52,632 - INFO - control 2026-08-31 02:25:52,637 - INFO - control-nodemgr:active 2026-08-31 02:25:52,642 - INFO - control:active 2026-08-31 02:25:52,643 - INFO - named:active 2026-08-31 02:25:52,649 - INFO - dns:active 2026-08-31 02:25:52,650 - INFO - analytics 2026-08-31 02:25:52,655 - INFO - analytics-nodemgr:active 2026-08-31 02:25:52,660 - INFO - analytics-api:active 2026-08-31 02:25:52,666 - INFO - collector:active 2026-08-31 02:25:52,666 - INFO - analytics-database 2026-08-31 02:25:52,666 - INFO - analytics-cassandra:active 2026-08-31 02:25:52,672 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:25:52,680 - INFO - query-engine:active 2026-08-31 02:25:52,680 - INFO - webui 2026-08-31 02:25:52,680 - INFO - webui:active 2026-08-31 02:25:52,680 - INFO - webui-middleware:active 2026-08-31 02:25:52,680 - INFO - redis:active 2026-08-31 02:25:52,680 - INFO - analytics_snmp 2026-08-31 02:25:52,685 - INFO - snmp-collector:active 2026-08-31 02:25:52,685 - INFO - snmp-topology:active 2026-08-31 02:25:52,685 - INFO - analytics_alarm 2026-08-31 02:25:52,685 - INFO - alarmgen:active 2026-08-31 02:25:52,741 - INFO - 10.0.0.186 2026-08-31 02:25:52,741 - INFO - vrouter 2026-08-31 02:25:52,747 - INFO - vrouter-nodemgr:active 2026-08-31 02:25:52,753 - INFO - agent:active 2026-08-31 02:25:52,809 - INFO - 10.0.0.176 2026-08-31 02:25:52,809 - INFO - vrouter 2026-08-31 02:25:52,815 - INFO - vrouter-nodemgr:active 2026-08-31 02:25:52,821 - INFO - agent:active 2026-08-31 02:25:52,821 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down'}}}) 2026-08-31 02:25:52,821 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 1 2026-08-31 02:26:02,899 - INFO - 10.0.0.50 2026-08-31 02:26:02,899 - INFO - config 2026-08-31 02:26:02,907 - INFO - config-nodemgr:active 2026-08-31 02:26:02,914 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down) 2026-08-31 02:26:02,920 - INFO - schema:active 2026-08-31 02:26:02,925 - INFO - svc-monitor:active 2026-08-31 02:26:02,930 - INFO - device-manager:active 2026-08-31 02:26:02,930 - INFO - config-database 2026-08-31 02:26:02,930 - INFO - config-cassandra:active 2026-08-31 02:26:02,931 - INFO - config-zookeeper:active 2026-08-31 02:26:02,931 - INFO - config-rabbitmq:active 2026-08-31 02:26:02,931 - INFO - control 2026-08-31 02:26:02,938 - INFO - control-nodemgr:active 2026-08-31 02:26:02,946 - INFO - control:active 2026-08-31 02:26:02,946 - INFO - named:active 2026-08-31 02:26:02,953 - INFO - dns:active 2026-08-31 02:26:02,953 - INFO - analytics 2026-08-31 02:26:02,957 - INFO - analytics-nodemgr:active 2026-08-31 02:26:02,962 - INFO - analytics-api:active 2026-08-31 02:26:02,969 - INFO - collector:active 2026-08-31 02:26:02,970 - INFO - analytics-database 2026-08-31 02:26:02,970 - INFO - analytics-cassandra:active 2026-08-31 02:26:02,974 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:26:02,982 - INFO - query-engine:active 2026-08-31 02:26:02,982 - INFO - webui 2026-08-31 02:26:02,982 - INFO - webui:active 2026-08-31 02:26:02,982 - INFO - webui-middleware:active 2026-08-31 02:26:02,982 - INFO - redis:active 2026-08-31 02:26:02,982 - INFO - analytics_snmp 2026-08-31 02:26:02,986 - INFO - snmp-collector:active 2026-08-31 02:26:02,987 - INFO - snmp-topology:active 2026-08-31 02:26:02,987 - INFO - analytics_alarm 2026-08-31 02:26:02,987 - INFO - alarmgen:active 2026-08-31 02:26:03,041 - INFO - 10.0.0.185 2026-08-31 02:26:03,042 - INFO - config 2026-08-31 02:26:03,047 - INFO - config-nodemgr:active 2026-08-31 02:26:03,052 - INFO - api-server:active 2026-08-31 02:26:03,057 - INFO - schema:backup 2026-08-31 02:26:03,062 - INFO - svc-monitor:backup 2026-08-31 02:26:03,069 - INFO - device-manager:backup 2026-08-31 02:26:03,069 - INFO - config-database 2026-08-31 02:26:03,069 - INFO - config-cassandra:active 2026-08-31 02:26:03,069 - INFO - config-zookeeper:active 2026-08-31 02:26:03,069 - INFO - config-rabbitmq:active 2026-08-31 02:26:03,069 - INFO - control 2026-08-31 02:26:03,075 - INFO - control-nodemgr:active 2026-08-31 02:26:03,080 - INFO - control:active 2026-08-31 02:26:03,080 - INFO - named:active 2026-08-31 02:26:03,086 - INFO - dns:active 2026-08-31 02:26:03,086 - INFO - analytics 2026-08-31 02:26:03,092 - INFO - analytics-nodemgr:active 2026-08-31 02:26:03,098 - INFO - analytics-api:active 2026-08-31 02:26:03,104 - INFO - collector:active 2026-08-31 02:26:03,104 - INFO - analytics-database 2026-08-31 02:26:03,104 - INFO - analytics-cassandra:active 2026-08-31 02:26:03,110 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:26:03,117 - INFO - query-engine:active 2026-08-31 02:26:03,117 - INFO - webui 2026-08-31 02:26:03,117 - INFO - webui:active 2026-08-31 02:26:03,117 - INFO - webui-middleware:active 2026-08-31 02:26:03,117 - INFO - redis:active 2026-08-31 02:26:03,117 - INFO - analytics_snmp 2026-08-31 02:26:03,122 - INFO - snmp-collector:active 2026-08-31 02:26:03,122 - INFO - snmp-topology:active 2026-08-31 02:26:03,122 - INFO - analytics_alarm 2026-08-31 02:26:03,122 - INFO - alarmgen:active 2026-08-31 02:26:03,188 - INFO - 10.0.0.38 2026-08-31 02:26:03,188 - INFO - config 2026-08-31 02:26:03,194 - INFO - config-nodemgr:active 2026-08-31 02:26:03,199 - INFO - api-server:active 2026-08-31 02:26:03,203 - INFO - schema:backup 2026-08-31 02:26:03,206 - INFO - svc-monitor:backup 2026-08-31 02:26:03,210 - INFO - device-manager:backup 2026-08-31 02:26:03,210 - INFO - config-database 2026-08-31 02:26:03,211 - INFO - config-cassandra:active 2026-08-31 02:26:03,211 - INFO - config-zookeeper:active 2026-08-31 02:26:03,211 - INFO - config-rabbitmq:active 2026-08-31 02:26:03,211 - INFO - control 2026-08-31 02:26:03,216 - INFO - control-nodemgr:active 2026-08-31 02:26:03,224 - INFO - control:active 2026-08-31 02:26:03,224 - INFO - named:active 2026-08-31 02:26:03,230 - INFO - dns:active 2026-08-31 02:26:03,230 - INFO - analytics 2026-08-31 02:26:03,236 - INFO - analytics-nodemgr:active 2026-08-31 02:26:03,243 - INFO - analytics-api:active 2026-08-31 02:26:03,250 - INFO - collector:active 2026-08-31 02:26:03,251 - INFO - analytics-database 2026-08-31 02:26:03,252 - INFO - analytics-cassandra:active 2026-08-31 02:26:03,258 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:26:03,265 - INFO - query-engine:active 2026-08-31 02:26:03,266 - INFO - webui 2026-08-31 02:26:03,266 - INFO - webui:active 2026-08-31 02:26:03,266 - INFO - webui-middleware:active 2026-08-31 02:26:03,267 - INFO - redis:active 2026-08-31 02:26:03,267 - INFO - analytics_snmp 2026-08-31 02:26:03,274 - INFO - snmp-collector:active 2026-08-31 02:26:03,275 - INFO - snmp-topology:active 2026-08-31 02:26:03,276 - INFO - analytics_alarm 2026-08-31 02:26:03,276 - INFO - alarmgen:active 2026-08-31 02:26:03,334 - INFO - 10.0.0.186 2026-08-31 02:26:03,334 - INFO - vrouter 2026-08-31 02:26:03,341 - INFO - vrouter-nodemgr:active 2026-08-31 02:26:03,347 - INFO - agent:active 2026-08-31 02:26:03,393 - INFO - 10.0.0.176 2026-08-31 02:26:03,393 - INFO - vrouter 2026-08-31 02:26:03,399 - INFO - vrouter-nodemgr:active 2026-08-31 02:26:03,405 - INFO - agent:active 2026-08-31 02:26:03,405 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down'}}}) 2026-08-31 02:26:03,405 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 2 2026-08-31 02:26:13,485 - INFO - 10.0.0.50 2026-08-31 02:26:13,485 - INFO - config 2026-08-31 02:26:13,492 - INFO - config-nodemgr:active 2026-08-31 02:26:13,497 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down) 2026-08-31 02:26:13,502 - INFO - schema:active 2026-08-31 02:26:13,507 - INFO - svc-monitor:active 2026-08-31 02:26:13,511 - INFO - device-manager:active 2026-08-31 02:26:13,511 - INFO - config-database 2026-08-31 02:26:13,511 - INFO - config-cassandra:active 2026-08-31 02:26:13,512 - INFO - config-zookeeper:active 2026-08-31 02:26:13,512 - INFO - config-rabbitmq:active 2026-08-31 02:26:13,512 - INFO - control 2026-08-31 02:26:13,517 - INFO - control-nodemgr:active 2026-08-31 02:26:13,527 - INFO - control:active 2026-08-31 02:26:13,527 - INFO - named:active 2026-08-31 02:26:13,535 - INFO - dns:active 2026-08-31 02:26:13,535 - INFO - analytics 2026-08-31 02:26:13,540 - INFO - analytics-nodemgr:active 2026-08-31 02:26:13,544 - INFO - analytics-api:active 2026-08-31 02:26:13,551 - INFO - collector:active 2026-08-31 02:26:13,551 - INFO - analytics-database 2026-08-31 02:26:13,551 - INFO - analytics-cassandra:active 2026-08-31 02:26:13,556 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:26:13,565 - INFO - query-engine:active 2026-08-31 02:26:13,565 - INFO - webui 2026-08-31 02:26:13,565 - INFO - webui:active 2026-08-31 02:26:13,566 - INFO - webui-middleware:active 2026-08-31 02:26:13,566 - INFO - redis:active 2026-08-31 02:26:13,566 - INFO - analytics_snmp 2026-08-31 02:26:13,570 - INFO - snmp-collector:active 2026-08-31 02:26:13,570 - INFO - snmp-topology:active 2026-08-31 02:26:13,570 - INFO - analytics_alarm 2026-08-31 02:26:13,570 - INFO - alarmgen:active 2026-08-31 02:26:13,616 - INFO - 10.0.0.185 2026-08-31 02:26:13,616 - INFO - config 2026-08-31 02:26:13,622 - INFO - config-nodemgr:active 2026-08-31 02:26:13,627 - INFO - api-server:active 2026-08-31 02:26:13,631 - INFO - schema:backup 2026-08-31 02:26:13,634 - INFO - svc-monitor:backup 2026-08-31 02:26:13,637 - INFO - device-manager:backup 2026-08-31 02:26:13,637 - INFO - config-database 2026-08-31 02:26:13,637 - INFO - config-cassandra:active 2026-08-31 02:26:13,638 - INFO - config-zookeeper:active 2026-08-31 02:26:13,638 - INFO - config-rabbitmq:active 2026-08-31 02:26:13,638 - INFO - control 2026-08-31 02:26:13,643 - INFO - control-nodemgr:active 2026-08-31 02:26:13,650 - INFO - control:active 2026-08-31 02:26:13,650 - INFO - named:active 2026-08-31 02:26:13,655 - INFO - dns:active 2026-08-31 02:26:13,655 - INFO - analytics 2026-08-31 02:26:13,659 - INFO - analytics-nodemgr:active 2026-08-31 02:26:13,664 - INFO - analytics-api:active 2026-08-31 02:26:13,669 - INFO - collector:active 2026-08-31 02:26:13,669 - INFO - analytics-database 2026-08-31 02:26:13,669 - INFO - analytics-cassandra:active 2026-08-31 02:26:13,674 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:26:13,682 - INFO - query-engine:active 2026-08-31 02:26:13,683 - INFO - webui 2026-08-31 02:26:13,683 - INFO - webui:active 2026-08-31 02:26:13,683 - INFO - webui-middleware:active 2026-08-31 02:26:13,683 - INFO - redis:active 2026-08-31 02:26:13,683 - INFO - analytics_snmp 2026-08-31 02:26:13,689 - INFO - snmp-collector:active 2026-08-31 02:26:13,689 - INFO - snmp-topology:active 2026-08-31 02:26:13,689 - INFO - analytics_alarm 2026-08-31 02:26:13,689 - INFO - alarmgen:active 2026-08-31 02:26:13,755 - INFO - 10.0.0.38 2026-08-31 02:26:13,756 - INFO - config 2026-08-31 02:26:13,761 - INFO - config-nodemgr:active 2026-08-31 02:26:13,767 - INFO - api-server:active 2026-08-31 02:26:13,770 - INFO - schema:backup 2026-08-31 02:26:13,774 - INFO - svc-monitor:backup 2026-08-31 02:26:13,777 - INFO - device-manager:backup 2026-08-31 02:26:13,777 - INFO - config-database 2026-08-31 02:26:13,777 - INFO - config-cassandra:active 2026-08-31 02:26:13,777 - INFO - config-zookeeper:active 2026-08-31 02:26:13,777 - INFO - config-rabbitmq:active 2026-08-31 02:26:13,777 - INFO - control 2026-08-31 02:26:13,783 - INFO - control-nodemgr:active 2026-08-31 02:26:13,791 - INFO - control:active 2026-08-31 02:26:13,791 - INFO - named:active 2026-08-31 02:26:13,798 - INFO - dns:active 2026-08-31 02:26:13,799 - INFO - analytics 2026-08-31 02:26:13,804 - INFO - analytics-nodemgr:active 2026-08-31 02:26:13,810 - INFO - analytics-api:active 2026-08-31 02:26:13,815 - INFO - collector:active 2026-08-31 02:26:13,815 - INFO - analytics-database 2026-08-31 02:26:13,815 - INFO - analytics-cassandra:active 2026-08-31 02:26:13,821 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:26:13,856 - INFO - query-engine:active 2026-08-31 02:26:13,856 - INFO - webui 2026-08-31 02:26:13,857 - INFO - webui:active 2026-08-31 02:26:13,857 - INFO - webui-middleware:active 2026-08-31 02:26:13,857 - INFO - redis:active 2026-08-31 02:26:13,857 - INFO - analytics_snmp 2026-08-31 02:26:13,862 - INFO - snmp-collector:active 2026-08-31 02:26:13,862 - INFO - snmp-topology:active 2026-08-31 02:26:13,862 - INFO - analytics_alarm 2026-08-31 02:26:13,862 - INFO - alarmgen:active 2026-08-31 02:26:13,929 - INFO - 10.0.0.186 2026-08-31 02:26:13,929 - INFO - vrouter 2026-08-31 02:26:13,934 - INFO - vrouter-nodemgr:active 2026-08-31 02:26:13,939 - INFO - agent:active 2026-08-31 02:26:13,996 - INFO - 10.0.0.176 2026-08-31 02:26:13,996 - INFO - vrouter 2026-08-31 02:26:14,001 - INFO - vrouter-nodemgr:active 2026-08-31 02:26:14,008 - INFO - agent:active 2026-08-31 02:26:14,008 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down'}}}) 2026-08-31 02:26:14,008 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 3 2026-08-31 02:26:24,075 - INFO - 10.0.0.50 2026-08-31 02:26:24,075 - INFO - config 2026-08-31 02:26:24,080 - INFO - config-nodemgr:active 2026-08-31 02:26:24,085 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down) 2026-08-31 02:26:24,089 - INFO - schema:active 2026-08-31 02:26:24,093 - INFO - svc-monitor:active 2026-08-31 02:26:24,097 - INFO - device-manager:active 2026-08-31 02:26:24,097 - INFO - config-database 2026-08-31 02:26:24,097 - INFO - config-cassandra:active 2026-08-31 02:26:24,097 - INFO - config-zookeeper:active 2026-08-31 02:26:24,097 - INFO - config-rabbitmq:active 2026-08-31 02:26:24,097 - INFO - control 2026-08-31 02:26:24,103 - INFO - control-nodemgr:active 2026-08-31 02:26:24,109 - INFO - control:active 2026-08-31 02:26:24,109 - INFO - named:active 2026-08-31 02:26:24,116 - INFO - dns:active 2026-08-31 02:26:24,117 - INFO - analytics 2026-08-31 02:26:24,122 - INFO - analytics-nodemgr:active 2026-08-31 02:26:24,127 - INFO - analytics-api:active 2026-08-31 02:26:24,134 - INFO - collector:active 2026-08-31 02:26:24,134 - INFO - analytics-database 2026-08-31 02:26:24,134 - INFO - analytics-cassandra:active 2026-08-31 02:26:24,140 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:26:24,160 - INFO - query-engine:active 2026-08-31 02:26:24,160 - INFO - webui 2026-08-31 02:26:24,160 - INFO - webui:active 2026-08-31 02:26:24,161 - INFO - webui-middleware:active 2026-08-31 02:26:24,161 - INFO - redis:active 2026-08-31 02:26:24,161 - INFO - analytics_snmp 2026-08-31 02:26:24,166 - INFO - snmp-collector:active 2026-08-31 02:26:24,166 - INFO - snmp-topology:active 2026-08-31 02:26:24,166 - INFO - analytics_alarm 2026-08-31 02:26:24,166 - INFO - alarmgen:active 2026-08-31 02:26:24,222 - INFO - 10.0.0.185 2026-08-31 02:26:24,222 - INFO - config 2026-08-31 02:26:24,227 - INFO - config-nodemgr:active 2026-08-31 02:26:24,232 - INFO - api-server:active 2026-08-31 02:26:24,235 - INFO - schema:backup 2026-08-31 02:26:24,238 - INFO - svc-monitor:backup 2026-08-31 02:26:24,242 - INFO - device-manager:backup 2026-08-31 02:26:24,242 - INFO - config-database 2026-08-31 02:26:24,242 - INFO - config-cassandra:active 2026-08-31 02:26:24,242 - INFO - config-zookeeper:active 2026-08-31 02:26:24,242 - INFO - config-rabbitmq:active 2026-08-31 02:26:24,242 - INFO - control 2026-08-31 02:26:24,247 - INFO - control-nodemgr:active 2026-08-31 02:26:24,254 - INFO - control:active 2026-08-31 02:26:24,254 - INFO - named:active 2026-08-31 02:26:24,259 - INFO - dns:active 2026-08-31 02:26:24,259 - INFO - analytics 2026-08-31 02:26:24,264 - INFO - analytics-nodemgr:active 2026-08-31 02:26:24,268 - INFO - analytics-api:active 2026-08-31 02:26:24,273 - INFO - collector:active 2026-08-31 02:26:24,273 - INFO - analytics-database 2026-08-31 02:26:24,273 - INFO - analytics-cassandra:active 2026-08-31 02:26:24,277 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:26:24,285 - INFO - query-engine:active 2026-08-31 02:26:24,286 - INFO - webui 2026-08-31 02:26:24,286 - INFO - webui:active 2026-08-31 02:26:24,286 - INFO - webui-middleware:active 2026-08-31 02:26:24,286 - INFO - redis:active 2026-08-31 02:26:24,286 - INFO - analytics_snmp 2026-08-31 02:26:24,290 - INFO - snmp-collector:active 2026-08-31 02:26:24,290 - INFO - snmp-topology:active 2026-08-31 02:26:24,290 - INFO - analytics_alarm 2026-08-31 02:26:24,290 - INFO - alarmgen:active 2026-08-31 02:26:24,346 - INFO - 10.0.0.38 2026-08-31 02:26:24,346 - INFO - config 2026-08-31 02:26:24,353 - INFO - config-nodemgr:active 2026-08-31 02:26:24,358 - INFO - api-server:active 2026-08-31 02:26:24,361 - INFO - schema:backup 2026-08-31 02:26:24,364 - INFO - svc-monitor:backup 2026-08-31 02:26:24,368 - INFO - device-manager:backup 2026-08-31 02:26:24,368 - INFO - config-database 2026-08-31 02:26:24,368 - INFO - config-cassandra:active 2026-08-31 02:26:24,368 - INFO - config-zookeeper:active 2026-08-31 02:26:24,368 - INFO - config-rabbitmq:active 2026-08-31 02:26:24,368 - INFO - control 2026-08-31 02:26:24,374 - INFO - control-nodemgr:active 2026-08-31 02:26:24,378 - INFO - control:active 2026-08-31 02:26:24,379 - INFO - named:active 2026-08-31 02:26:24,384 - INFO - dns:active 2026-08-31 02:26:24,384 - INFO - analytics 2026-08-31 02:26:24,389 - INFO - analytics-nodemgr:active 2026-08-31 02:26:24,393 - INFO - analytics-api:active 2026-08-31 02:26:24,399 - INFO - collector:active 2026-08-31 02:26:24,399 - INFO - analytics-database 2026-08-31 02:26:24,399 - INFO - analytics-cassandra:active 2026-08-31 02:26:24,404 - INFO - analyticsdb-nodemgr:active 2026-08-31 02:26:24,420 - INFO - query-engine:active 2026-08-31 02:26:24,420 - INFO - webui 2026-08-31 02:26:24,421 - INFO - webui:active 2026-08-31 02:26:24,421 - INFO - webui-middleware:active 2026-08-31 02:26:24,421 - INFO - redis:active 2026-08-31 02:26:24,421 - INFO - analytics_snmp 2026-08-31 02:26:24,425 - INFO - snmp-collector:active 2026-08-31 02:26:24,425 - INFO - snmp-topology:active 2026-08-31 02:26:24,425 - INFO - analytics_alarm 2026-08-31 02:26:24,425 - INFO - alarmgen:active 2026-08-31 02:26:24,481 - INFO - 10.0.0.186 2026-08-31 02:26:24,481 - INFO - vrouter 2026-08-31 02:26:24,487 - INFO - vrouter-nodemgr:active 2026-08-31 02:26:24,492 - INFO - agent:active 2026-08-31 02:26:24,548 - INFO - 10.0.0.176 2026-08-31 02:26:24,548 - INFO - vrouter 2026-08-31 02:26:24,555 - INFO - vrouter-nodemgr:active 2026-08-31 02:26:24,561 - INFO - agent:active 2026-08-31 02:26:24,562 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down'}}}) 2026-08-31 02:26:24,562 - ERROR - Not all services up , Gave up! 2026-08-31 02:26:24,562 - INFO - Failed services are : defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-08-31 02:25:40.892690] connection down'}}}) 2026-08-31 02:26:24,562 - ERROR - contrail-status failed 2026-08-31 02:26:25,244 - DEBUG - No XMPP flaps were noticed during the test 2026-08-31 02:26:25,244 - INFO - 2026-08-31 02:26:25,245 - INFO - END TEST : test_contrail_status : FAILED[0:00:44] 2026-08-31 02:26:25,245 - INFO - -------------------------------------------------------------------------------- 2026-08-31 04:43:36,386 - INFO - Using existing project ['default-domain', 'admin'](d21f6d7f-753d-4756-9c08-92953335e433) 2026-08-31 04:43:36,408 - INFO - ================================================================================ 2026-08-31 04:43:36,409 - INFO - STARTING TEST : test_contrail_status 2026-08-31 04:43:36,409 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2026-08-31 04:43:37,720 - DEBUG - Skipping xmpp flap check 2026-08-31 04:43:37,720 - INFO - Initial checks done. Running the testcase now 2026-08-31 04:43:37,720 - INFO - 2026-08-31 04:43:38,018 - INFO - 10.0.0.50 2026-08-31 04:43:38,018 - INFO - config 2026-08-31 04:43:38,026 - INFO - config-nodemgr:active 2026-08-31 04:43:38,033 - INFO - api-server:active 2026-08-31 04:43:38,039 - INFO - schema:active 2026-08-31 04:43:38,043 - INFO - svc-monitor:active 2026-08-31 04:43:38,048 - INFO - device-manager:active 2026-08-31 04:43:38,048 - INFO - config-database 2026-08-31 04:43:38,048 - INFO - config-cassandra:active 2026-08-31 04:43:38,048 - INFO - config-zookeeper:active 2026-08-31 04:43:38,048 - INFO - config-rabbitmq:active 2026-08-31 04:43:38,048 - INFO - control 2026-08-31 04:43:38,054 - INFO - control-nodemgr:active 2026-08-31 04:43:38,067 - INFO - control:active 2026-08-31 04:43:38,067 - INFO - named:active 2026-08-31 04:43:38,081 - INFO - dns:active 2026-08-31 04:43:38,081 - INFO - analytics 2026-08-31 04:43:38,089 - INFO - analytics-nodemgr:active 2026-08-31 04:43:38,097 - INFO - analytics-api:active 2026-08-31 04:43:38,113 - INFO - collector:active 2026-08-31 04:43:38,113 - INFO - analytics-database 2026-08-31 04:43:38,113 - INFO - analytics-cassandra:active 2026-08-31 04:43:38,119 - INFO - analyticsdb-nodemgr:active 2026-08-31 04:43:38,141 - INFO - query-engine:active 2026-08-31 04:43:38,141 - INFO - webui 2026-08-31 04:43:38,141 - INFO - webui:active 2026-08-31 04:43:38,141 - INFO - webui-middleware:active 2026-08-31 04:43:38,141 - INFO - redis:active 2026-08-31 04:43:38,141 - INFO - analytics_snmp 2026-08-31 04:43:38,146 - INFO - snmp-collector:active 2026-08-31 04:43:38,146 - INFO - snmp-topology:active 2026-08-31 04:43:38,146 - INFO - analytics_alarm 2026-08-31 04:43:38,146 - INFO - alarmgen:active 2026-08-31 04:43:38,248 - INFO - 10.0.0.185 2026-08-31 04:43:38,248 - INFO - config 2026-08-31 04:43:38,254 - INFO - config-nodemgr:active 2026-08-31 04:43:38,259 - INFO - api-server:active 2026-08-31 04:43:38,263 - INFO - schema:backup 2026-08-31 04:43:38,266 - INFO - svc-monitor:backup 2026-08-31 04:43:38,270 - INFO - device-manager:backup 2026-08-31 04:43:38,270 - INFO - config-database 2026-08-31 04:43:38,270 - INFO - config-cassandra:active 2026-08-31 04:43:38,270 - INFO - config-zookeeper:active 2026-08-31 04:43:38,270 - INFO - config-rabbitmq:active 2026-08-31 04:43:38,270 - INFO - control 2026-08-31 04:43:38,275 - INFO - control-nodemgr:active 2026-08-31 04:43:38,280 - INFO - control:active 2026-08-31 04:43:38,280 - INFO - named:active 2026-08-31 04:43:38,286 - INFO - dns:active 2026-08-31 04:43:38,286 - INFO - analytics 2026-08-31 04:43:38,291 - INFO - analytics-nodemgr:active 2026-08-31 04:43:38,298 - INFO - analytics-api:active 2026-08-31 04:43:38,303 - INFO - collector:active 2026-08-31 04:43:38,303 - INFO - analytics-database 2026-08-31 04:43:38,303 - INFO - analytics-cassandra:active 2026-08-31 04:43:38,310 - INFO - analyticsdb-nodemgr:active 2026-08-31 04:43:38,315 - INFO - query-engine:active 2026-08-31 04:43:38,315 - INFO - webui 2026-08-31 04:43:38,315 - INFO - webui:active 2026-08-31 04:43:38,315 - INFO - webui-middleware:active 2026-08-31 04:43:38,316 - INFO - redis:active 2026-08-31 04:43:38,316 - INFO - analytics_snmp 2026-08-31 04:43:38,320 - INFO - snmp-collector:active 2026-08-31 04:43:38,320 - INFO - snmp-topology:active 2026-08-31 04:43:38,320 - INFO - analytics_alarm 2026-08-31 04:43:38,320 - INFO - alarmgen:active 2026-08-31 04:43:38,430 - INFO - 10.0.0.38 2026-08-31 04:43:38,430 - INFO - config 2026-08-31 04:43:38,438 - INFO - config-nodemgr:active 2026-08-31 04:43:38,444 - INFO - api-server:active 2026-08-31 04:43:38,448 - INFO - schema:backup 2026-08-31 04:43:38,452 - INFO - svc-monitor:backup 2026-08-31 04:43:38,455 - INFO - device-manager:backup 2026-08-31 04:43:38,456 - INFO - config-database 2026-08-31 04:43:38,456 - INFO - config-cassandra:active 2026-08-31 04:43:38,456 - INFO - config-zookeeper:active 2026-08-31 04:43:38,456 - INFO - config-rabbitmq:active 2026-08-31 04:43:38,456 - INFO - control 2026-08-31 04:43:38,461 - INFO - control-nodemgr:active 2026-08-31 04:43:38,467 - INFO - control:active 2026-08-31 04:43:38,467 - INFO - named:active 2026-08-31 04:43:38,472 - INFO - dns:active 2026-08-31 04:43:38,472 - INFO - analytics 2026-08-31 04:43:38,479 - INFO - analytics-nodemgr:active 2026-08-31 04:43:38,485 - INFO - analytics-api:active 2026-08-31 04:43:38,492 - INFO - collector:active 2026-08-31 04:43:38,492 - INFO - analytics-database 2026-08-31 04:43:38,492 - INFO - analytics-cassandra:active 2026-08-31 04:43:38,499 - INFO - analyticsdb-nodemgr:active 2026-08-31 04:43:38,508 - INFO - query-engine:active 2026-08-31 04:43:38,508 - INFO - webui 2026-08-31 04:43:38,508 - INFO - webui:active 2026-08-31 04:43:38,508 - INFO - webui-middleware:active 2026-08-31 04:43:38,508 - INFO - redis:active 2026-08-31 04:43:38,508 - INFO - analytics_snmp 2026-08-31 04:43:38,515 - INFO - snmp-collector:active 2026-08-31 04:43:38,515 - INFO - snmp-topology:active 2026-08-31 04:43:38,515 - INFO - analytics_alarm 2026-08-31 04:43:38,515 - INFO - alarmgen:active 2026-08-31 04:43:38,617 - INFO - 10.0.0.186 2026-08-31 04:43:38,617 - INFO - vrouter 2026-08-31 04:43:38,624 - INFO - vrouter-nodemgr:active 2026-08-31 04:43:38,632 - INFO - agent:active 2026-08-31 04:43:38,723 - INFO - 10.0.0.176 2026-08-31 04:43:38,723 - INFO - vrouter 2026-08-31 04:43:38,730 - INFO - vrouter-nodemgr:active 2026-08-31 04:43:38,736 - INFO - agent:active 2026-08-31 04:43:38,736 - INFO - All the contrail services are up on all nodes 2026-08-31 04:43:38,736 - INFO - contrail-status passed 2026-08-31 04:43:40,043 - DEBUG - Skipping xmpp flap check 2026-08-31 04:43:40,043 - INFO - END TEST : test_contrail_status : PASSED[0:00:04] 2026-08-31 04:43:40,043 - INFO - --------------------------------------------------------------------------------