2025-08-14 03:47:12,873 - INFO - Domain Default found not creating 2025-08-14 03:47:13,003 - INFO - Project ctest-TestEvpnCasesRestart-59825578 not found, creating it 2025-08-14 03:47:13,495 - INFO - Created Project:ctest-TestEvpnCasesRestart-59825578, ID : 267caa7c-ab72-42d1-a420-7d6ef32f5c83 2025-08-14 03:47:14,873 - INFO - ================================================================================ 2025-08-14 03:47:14,873 - INFO - STARTING TEST : test_with_vxlan_encap_agent_restart 2025-08-14 03:47:14,873 - INFO - TEST DESCRIPTION : Description:Test agent restart with VXLAN Encap 1. Configure VXLAN as highest priority 2. Configure 2 VM under a VN configured with l2-l3 mode 3. Send L2 traffic and verify communication between vms 4. Restart the contrail-grouter service. 5. Send L2 Traffic and verify communication between vms Pass criteria: Step 3 and 5 should pass Maintainer: chhandak@juniper.net 2025-08-14 03:47:16,132 - DEBUG - Skipping xmpp flap check 2025-08-14 03:47:16,132 - INFO - Initial checks done. Running the testcase now 2025-08-14 03:47:16,132 - INFO - 2025-08-14 03:47:16,132 - INFO - Setting new Encap before continuing 2025-08-14 03:47:16,132 - INFO - Read the existing encap priority 2025-08-14 03:47:16,648 - INFO - Created.UUID is {"global-vrouter-config": {"uuid": "76c07a54-5d54-4e8f-a852-93ac202b7343", "href": "http://10.0.0.23:8082/global-vrouter-config/76c07a54-5d54-4e8f-a852-93ac202b7343"}}. VXLAN is the highest priority encap 2025-08-14 03:47:16,670 - DEBUG - Services list from nova: [, , , ] 2025-08-14 03:47:16,670 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3851-2', 'an-jenkins-deploy-platform-ansible-os-3851-1']} 2025-08-14 03:47:17,057 - DEBUG - Response for create_network : {'network': {'id': 'a9e32396-5694-4484-8edf-cb76eba7705d', 'name': 'EVPN-VN1', 'tenant_id': '267caa7cab7242d1a4207d6ef32f5c83', 'project_id': '267caa7cab7242d1a4207d6ef32f5c83', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestEvpnCasesRestart-59825578', 'EVPN-VN1'], 'port_security_enabled': True, 'description': ''}} 2025-08-14 03:47:17,290 - DEBUG - Response for create_subnet : {'subnet': {'id': '9bd51388-6137-4623-a9b6-3f37b641d3a8', 'name': '', 'tenant_id': '267caa7cab7242d1a4207d6ef32f5c83', 'network_id': 'a9e32396-5694-4484-8edf-cb76eba7705d', 'ip_version': 4, 'cidr': '11.1.1.0/24', 'allocation_pools': [{'start': '11.1.1.2', 'end': '11.1.1.254'}], 'gateway_ip': '11.1.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '11.1.1.2', 'tags': [], 'project_id': '267caa7cab7242d1a4207d6ef32f5c83'}} 2025-08-14 03:47:17,311 - INFO - Created VN EVPN-VN1 2025-08-14 03:47:17,325 - DEBUG - VN EVPN-VN1 UUID is a9e32396-5694-4484-8edf-cb76eba7705d 2025-08-14 03:47:18,521 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3851-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3851-2) 2025-08-14 03:47:19,794 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3851-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3851-1) 2025-08-14 03:47:19,794 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2025-08-14 03:47:20,108 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2025-08-14 03:47:20,158 - DEBUG - Requesting: http://10.0.0.38:8082/domain/16362ee5-4ea2-4ad6-8a9c-e34af37c4ada 2025-08-14 03:47:20,192 - DEBUG - Requesting: http://10.0.0.38:8082/project/267caa7c-ab72-42d1-a420-7d6ef32f5c83 2025-08-14 03:47:20,328 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/a9e32396-5694-4484-8edf-cb76eba7705d 2025-08-14 03:47:20,367 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/a9e32396-5694-4484-8edf-cb76eba7705d 2025-08-14 03:47:20,385 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/faec64a8-bd21-4b65-a564-9d0d9a920bcb 2025-08-14 03:47:20,400 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/faec64a8-bd21-4b65-a564-9d0d9a920bcb 2025-08-14 03:47:20,409 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/17595bf0-8c22-49c1-8bcd-8470161f7eda 2025-08-14 03:47:20,418 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-14 03:47:20,418 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/a9e32396-5694-4484-8edf-cb76eba7705d 2025-08-14 03:47:20,429 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/faec64a8-bd21-4b65-a564-9d0d9a920bcb 2025-08-14 03:47:20,445 - INFO - Verified VN network id 13 for VN a9e32396-5694-4484-8edf-cb76eba7705d 2025-08-14 03:47:20,446 - INFO - Verifications in API Server for VN EVPN-VN1 passed 2025-08-14 03:47:20,446 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/a9e32396-5694-4484-8edf-cb76eba7705d 2025-08-14 03:47:20,457 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/faec64a8-bd21-4b65-a564-9d0d9a920bcb 2025-08-14 03:47:20,466 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/faec64a8-bd21-4b65-a564-9d0d9a920bcb 2025-08-14 03:47:20,488 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/17595bf0-8c22-49c1-8bcd-8470161f7eda 2025-08-14 03:47:20,509 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestEvpnCasesRestart-59825578', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12241667340727764100', 'uuid-lslong': '10295170984812441693'}, 'enable': 'true', 'created': '2025-08-14T03:47:16', 'last-modified': '2025-08-14T03:47:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:03.270533'} 2025-08-14 03:47:20,525 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-14 03:47:20,535 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestEvpnCasesRestart-59825578', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12241667340727764100', 'uuid-lslong': '10295170984812441693'}, 'enable': 'true', 'created': '2025-08-14T03:47:16', 'last-modified': '2025-08-14T03:47:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:03.293264'} 2025-08-14 03:47:20,544 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-14 03:47:20,555 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestEvpnCasesRestart-59825578', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12241667340727764100', 'uuid-lslong': '10295170984812441693'}, 'enable': 'true', 'created': '2025-08-14T03:47:16', 'last-modified': '2025-08-14T03:47:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:03.309142'} 2025-08-14 03:47:20,571 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-14 03:47:20,571 - INFO - On all control nodes, Config, RI and RT verification for VN EVPN-VN1 passed 2025-08-14 03:47:20,571 - DEBUG - ====Verifying policy data for EVPN-VN1 in API_Server ====== 2025-08-14 03:47:20,571 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2025-08-14 03:47:20,587 - DEBUG - Requesting: http://10.0.0.38:8082/domain/16362ee5-4ea2-4ad6-8a9c-e34af37c4ada 2025-08-14 03:47:20,609 - DEBUG - Requesting: http://10.0.0.38:8082/project/267caa7c-ab72-42d1-a420-7d6ef32f5c83 2025-08-14 03:47:20,716 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/a9e32396-5694-4484-8edf-cb76eba7705d 2025-08-14 03:47:20,732 - DEBUG - =>VN EVPN-VN1 has no policy to be verified 2025-08-14 03:47:20,733 - DEBUG - Verifying the vn in opserver 2025-08-14 03:47:20,733 - DEBUG - Verifying the default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 virtual network link through opserver 10.0.0.38 2025-08-14 03:47:20,733 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-14 03:47:21,026 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-14 03:47:21,139 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1?flat'} 2025-08-14 03:47:21,139 - DEBUG - Verifying the default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 virtual network link through opserver 10.0.0.27 2025-08-14 03:47:21,139 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-14 03:47:21,864 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-14 03:47:22,040 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1?flat'} 2025-08-14 03:47:22,040 - DEBUG - Verifying the default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 virtual network link through opserver 10.0.0.23 2025-08-14 03:47:22,040 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-08-14 03:47:22,488 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-08-14 03:47:22,541 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1?flat'} 2025-08-14 03:47:22,542 - INFO - Validated that VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is found in opserver 2025-08-14 03:47:22,553 - DEBUG - VRF ids for VN EVPN-VN1: {'10.0.0.65': '2', '10.0.0.130': '2'} 2025-08-14 03:47:22,630 - DEBUG - VM is in ACTIVE state now 2025-08-14 03:47:22,630 - INFO - VM name : EVPN_VN1_VM1 2025-08-14 03:47:22,709 - DEBUG - VM EVPN_VN1_VM1 ID is f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:47:22,842 - DEBUG - VM is in ACTIVE state now 2025-08-14 03:47:22,868 - DEBUG - VM EVPN_VN1_VM1 launched on Node an-jenkins-deploy-platform-ansible-os-3851-2 2025-08-14 03:47:22,941 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:47:23,238 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:47:23,270 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:47:23,278 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ee63101f-4639-454c-b775-2ccd7ed65044 2025-08-14 03:47:23,292 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:47:23,301 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ee63101f-4639-454c-b775-2ccd7ed65044 2025-08-14 03:47:23,312 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/cbe64ad8-5d88-418b-a864-7436bb85523a 2025-08-14 03:47:23,323 - DEBUG - Verifying in api server 10.0.0.23 2025-08-14 03:47:23,323 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:47:23,331 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ee63101f-4639-454c-b775-2ccd7ed65044 2025-08-14 03:47:23,341 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/cbe64ad8-5d88-418b-a864-7436bb85523a 2025-08-14 03:47:23,350 - INFO - VM EVPN_VN1_VM1 verfication in all API Servers passed 2025-08-14 03:47:23,370 - DEBUG - VM EVPN_VN1_VM1 Tap interface: {'index': '3', 'name': 'tapee63101f-46', 'uuid': 'ee63101f-4639-454c-b775-2ccd7ed65044', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'vm_uuid': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'vm_name': 'EVPN_VN1_VM1', 'ip_addr': '11.1.1.3', 'mac_addr': '02:ee:63:10:1f:46', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '150', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044', 'sg_uuid_list': ['0acf68aa-9333-4001-a6ce-69695242eb74'], 'static_route_list': None, 'vm_project_uuid': '267caa7c-ab72-42d1-a420-7d6ef32f5c83', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-14 03:47:23,370 - DEBUG - Agent 10.0.0.130 vrf name: default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1 2025-08-14 03:47:23,376 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-08-14 03:47:23,463 - DEBUG - VM EVPN_VN1_VM1 : Tap interface tapee63101f-46 is set to Active 2025-08-14 03:47:23,463 - DEBUG - Tap interface tapee63101f-46 detail : {'index': '3', 'name': 'tapee63101f-46', 'uuid': 'ee63101f-4639-454c-b775-2ccd7ed65044', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'vm_uuid': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'vm_name': 'EVPN_VN1_VM1', 'ip_addr': '11.1.1.3', 'mac_addr': '02:ee:63:10:1f:46', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '150', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044', 'sg_uuid_list': ['0acf68aa-9333-4001-a6ce-69695242eb74'], 'static_route_list': None, 'vm_project_uuid': '267caa7c-ab72-42d1-a420-7d6ef32f5c83', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-14 03:47:23,577 - DEBUG - Starting Layer 2 verification in Agent 2025-08-14 03:47:23,589 - DEBUG - Layer 2 path is seen for VM MAC 02:ee:63:10:1f:46 in agent 10.0.0.130 2025-08-14 03:47:23,590 - DEBUG - Active layer 2 route in agent is present for VMI tapee63101f-46 2025-08-14 03:47:23,590 - DEBUG - vxlan_id (13) matches bw route table and interface table 2025-08-14 03:47:23,596 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is found in Agent of node 10.0.0.65 2025-08-14 03:47:23,731 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is consistent in agent 10.0.0.65 2025-08-14 03:47:23,732 - DEBUG - Route for VM IP ['11.1.1.3'] is consistent in agent 10.0.0.65 2025-08-14 03:47:23,732 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 verification for VM EVPN_VN1_VM1 in Agent 10.0.0.65 passed 2025-08-14 03:47:23,844 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2025-08-14 03:47:23,859 - DEBUG - Route for VM MAC 02:ee:63:10:1f:46 is consistent in agent 10.0.0.65 2025-08-14 03:47:23,866 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is found in Agent of node 10.0.0.130 2025-08-14 03:47:24,002 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is consistent in agent 10.0.0.130 2025-08-14 03:47:24,003 - DEBUG - Route for VM IP ['11.1.1.3'] is consistent in agent 10.0.0.130 2025-08-14 03:47:24,003 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 verification for VM EVPN_VN1_VM1 in Agent 10.0.0.130 passed 2025-08-14 03:47:24,120 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-08-14 03:47:24,135 - DEBUG - Route for VM MAC 02:ee:63:10:1f:46 is consistent in agent 10.0.0.130 2025-08-14 03:47:27,308 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1004ms') 2025-08-14 03:47:27,308 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-08-14 03:47:27,322 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-08-14 03:47:31,406 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-08-14 03:47:31,406 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-08-14 03:47:31,421 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-08-14 03:47:35,498 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-08-14 03:47:35,499 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-08-14 03:47:35,514 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-08-14 03:47:39,598 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms') 2025-08-14 03:47:39,599 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-08-14 03:47:39,612 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-08-14 03:47:43,688 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-08-14 03:47:43,688 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-08-14 03:47:43,704 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-08-14 03:47:47,792 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-08-14 03:47:47,792 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-08-14 03:47:47,808 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-08-14 03:47:51,881 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-08-14 03:47:51,882 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-08-14 03:47:51,894 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-08-14 03:47:53,960 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=6.79 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.52 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 1.522/4.158/6.794/2.636 ms') 2025-08-14 03:47:53,960 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-08-14 03:47:53,961 - INFO - VM EVPN_VN1_VM1 verifications in Compute nodes passed 2025-08-14 03:47:54,172 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.65 2025-08-14 03:47:54,175 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.192, 25; Got 10.20.0.192,25 2025-08-14 03:47:54,175 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.65 2025-08-14 03:47:54,293 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.130 2025-08-14 03:47:54,297 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-08-14 03:47:54,297 - INFO - Validated routes of VM EVPN_VN1_VM1 in all vrouters 2025-08-14 03:47:54,789 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-08-14 03:47:54,916 - DEBUG - Layer2 route found for VM MAC 02:ee:63:10:1f:46 in Control-node 10.0.0.38 2025-08-14 03:47:54,916 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.38 and Agent, Expected: 13, Seen: 13 2025-08-14 03:47:54,916 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-08-14 03:47:55,043 - DEBUG - Layer2 route found for VM MAC 02:ee:63:10:1f:46 in Control-node 10.0.0.27 2025-08-14 03:47:55,043 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.27 and Agent, Expected: 13, Seen: 13 2025-08-14 03:47:55,043 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-08-14 03:47:55,173 - DEBUG - Layer2 route found for VM MAC 02:ee:63:10:1f:46 in Control-node 10.0.0.23 2025-08-14 03:47:55,173 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.23 and Agent, Expected: 13, Seen: 13 2025-08-14 03:47:55,173 - INFO - Verification in Control-nodes for VM EVPN_VN1_VM1 passed 2025-08-14 03:47:55,173 - DEBUG - Verifying the vm in opserver 2025-08-14 03:47:55,173 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-08-14 03:47:55,173 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:47:55,239 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044?flat 2025-08-14 03:47:55,296 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044?flat 2025-08-14 03:47:55,359 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:47:55,415 - DEBUG - Verifying vm in vn uve 2025-08-14 03:47:55,415 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044?flat 2025-08-14 03:47:55,438 - DEBUG - VM uve shows interface as {'rx_vlan': 65535, 'virtual_network': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'port_mirror_enabled': False, 'label': 25, 'is_health_check_active': True, 'mac_address': '02:ee:63:10:1f:46', 'ip4_active': True, 'fixed_ip4_list': ['11.1.1.3'], 'uuid': 'ee63101f-4639-454c-b775-2ccd7ed65044', 'l2_active': True, 'ip6_active': False, 'tx_vlan': 65535, 'vm_name': 'EVPN_VN1_VM1', 'ip6_address': '::', 'gateway': '11.1.1.1', '__T': 1755143260156527, 'active': True, 'vhostuser_mode': 0, 'ip_address': '11.1.1.3', 'vn_uuid': 'a9e32396-5694-4484-8edf-cb76eba7705d', 'vm_uuid': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'admin_state': True} 2025-08-14 03:47:55,438 - DEBUG - VM uve shows ip address as ['11.1.1.3', '::'] 2025-08-14 03:47:55,438 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:47:55,438 - INFO - Verifying through opserver in 10.0.0.38 2025-08-14 03:47:55,438 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1?flat 2025-08-14 03:47:55,466 - DEBUG - expected vm list f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:47:55,466 - DEBUG - Extracted vm list ['f40f0938-5b54-4c7c-8ac7-14b360cbabc9'] 2025-08-14 03:47:55,466 - INFO - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is present in default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:47:55,466 - DEBUG - Verifying vm in vrouter uve 2025-08-14 03:47:55,466 - DEBUG - Getting info from collector 10.0.0.23.. 2025-08-14 03:47:55,466 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:47:55,482 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-3851-2'] 2025-08-14 03:47:55,482 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-2:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:47:55,535 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-2?flat 2025-08-14 03:47:55,613 - DEBUG - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:47:55,613 - DEBUG - Validated that VM EVPN_VN1_VM1 is in Vrouter an-jenkins-deploy-platform-ansible-os-3851-2 UVE 2025-08-14 03:47:55,613 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-08-14 03:47:55,621 - DEBUG - Expected tap interface of VM uuid f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044 2025-08-14 03:47:55,621 - DEBUG - Expected VN of VM uuid f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:47:55,621 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-2:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:47:55,673 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-2?flat 2025-08-14 03:47:55,748 - DEBUG - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:47:55,749 - INFO - tap interface default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044 of vm f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:47:55,749 - DEBUG - Connected network default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 of vm f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:47:55,749 - DEBUG - Verifying the f40f0938-5b54-4c7c-8ac7-14b360cbabc9 virtual network link through opserver 10.0.0.38 2025-08-14 03:47:55,749 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-14 03:47:55,761 - DEBUG - VM link and name as {'name': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat'} 2025-08-14 03:47:55,761 - DEBUG - Verifying the f40f0938-5b54-4c7c-8ac7-14b360cbabc9 virtual network link through opserver 10.0.0.27 2025-08-14 03:47:55,761 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-08-14 03:47:55,773 - DEBUG - VM link and name as {'name': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat'} 2025-08-14 03:47:55,773 - DEBUG - Verifying the f40f0938-5b54-4c7c-8ac7-14b360cbabc9 virtual network link through opserver 10.0.0.23 2025-08-14 03:47:55,773 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-08-14 03:47:55,786 - DEBUG - VM link and name as {'name': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat'} 2025-08-14 03:47:55,786 - INFO - VM EVPN_VN1_VM1 validations in Opserver passed 2025-08-14 03:47:55,858 - DEBUG - VM is in ACTIVE state now 2025-08-14 03:47:55,858 - INFO - VM name : EVPN_VN1_VM2 2025-08-14 03:47:55,984 - DEBUG - VM EVPN_VN1_VM2 ID is 5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:47:56,049 - DEBUG - VM is in ACTIVE state now 2025-08-14 03:47:56,049 - DEBUG - VM EVPN_VN1_VM2 launched on Node an-jenkins-deploy-platform-ansible-os-3851-1 2025-08-14 03:47:56,114 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:47:56,123 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:47:56,130 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c64668ad-82ee-4572-b7c2-d4b6c3059197 2025-08-14 03:47:56,139 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:47:56,146 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c64668ad-82ee-4572-b7c2-d4b6c3059197 2025-08-14 03:47:56,156 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/37b80726-66bd-4187-a2e2-cc3e526d8182 2025-08-14 03:47:56,165 - DEBUG - Verifying in api server 10.0.0.23 2025-08-14 03:47:56,165 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:47:56,171 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c64668ad-82ee-4572-b7c2-d4b6c3059197 2025-08-14 03:47:56,181 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/37b80726-66bd-4187-a2e2-cc3e526d8182 2025-08-14 03:47:56,188 - INFO - VM EVPN_VN1_VM2 verfication in all API Servers passed 2025-08-14 03:47:56,207 - DEBUG - VM EVPN_VN1_VM2 Tap interface: {'index': '3', 'name': 'tapc64668ad-82', 'uuid': 'c64668ad-82ee-4572-b7c2-d4b6c3059197', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'vm_uuid': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'vm_name': 'EVPN_VN1_VM2', 'ip_addr': '11.1.1.4', 'mac_addr': '02:c6:46:68:ad:82', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '112', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197', 'sg_uuid_list': ['0acf68aa-9333-4001-a6ce-69695242eb74'], 'static_route_list': None, 'vm_project_uuid': '267caa7c-ab72-42d1-a420-7d6ef32f5c83', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-14 03:47:56,207 - DEBUG - Agent 10.0.0.65 vrf name: default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1 2025-08-14 03:47:56,214 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.18:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-08-14 03:47:56,292 - DEBUG - VM EVPN_VN1_VM2 : Tap interface tapc64668ad-82 is set to Active 2025-08-14 03:47:56,293 - DEBUG - Tap interface tapc64668ad-82 detail : {'index': '3', 'name': 'tapc64668ad-82', 'uuid': 'c64668ad-82ee-4572-b7c2-d4b6c3059197', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'vm_uuid': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'vm_name': 'EVPN_VN1_VM2', 'ip_addr': '11.1.1.4', 'mac_addr': '02:c6:46:68:ad:82', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '112', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197', 'sg_uuid_list': ['0acf68aa-9333-4001-a6ce-69695242eb74'], 'static_route_list': None, 'vm_project_uuid': '267caa7c-ab72-42d1-a420-7d6ef32f5c83', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-14 03:47:56,408 - DEBUG - Starting Layer 2 verification in Agent 2025-08-14 03:47:56,420 - DEBUG - Layer 2 path is seen for VM MAC 02:c6:46:68:ad:82 in agent 10.0.0.65 2025-08-14 03:47:56,421 - DEBUG - Active layer 2 route in agent is present for VMI tapc64668ad-82 2025-08-14 03:47:56,421 - DEBUG - vxlan_id (13) matches bw route table and interface table 2025-08-14 03:47:56,429 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is found in Agent of node 10.0.0.65 2025-08-14 03:47:56,561 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is consistent in agent 10.0.0.65 2025-08-14 03:47:56,561 - DEBUG - Route for VM IP ['11.1.1.4'] is consistent in agent 10.0.0.65 2025-08-14 03:47:56,561 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 verification for VM EVPN_VN1_VM2 in Agent 10.0.0.65 passed 2025-08-14 03:47:56,676 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2025-08-14 03:47:56,689 - DEBUG - Route for VM MAC 02:c6:46:68:ad:82 is consistent in agent 10.0.0.65 2025-08-14 03:47:56,695 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is found in Agent of node 10.0.0.130 2025-08-14 03:47:56,826 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is consistent in agent 10.0.0.130 2025-08-14 03:47:56,826 - DEBUG - Route for VM IP ['11.1.1.4'] is consistent in agent 10.0.0.130 2025-08-14 03:47:56,826 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 verification for VM EVPN_VN1_VM2 in Agent 10.0.0.130 passed 2025-08-14 03:47:56,936 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-08-14 03:47:56,949 - DEBUG - Route for VM MAC 02:c6:46:68:ad:82 is consistent in agent 10.0.0.130 2025-08-14 03:47:58,122 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.61 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.37 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 3.365/4.485/5.606/1.120 ms') 2025-08-14 03:47:58,122 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM2 passed 2025-08-14 03:47:58,122 - INFO - VM EVPN_VN1_VM2 verifications in Compute nodes passed 2025-08-14 03:47:58,322 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.65 2025-08-14 03:47:58,325 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-08-14 03:47:58,450 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.130 2025-08-14 03:47:58,455 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.18, 25; Got 10.20.0.18,25 2025-08-14 03:47:58,455 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.130 2025-08-14 03:47:58,455 - INFO - Validated routes of VM EVPN_VN1_VM2 in all vrouters 2025-08-14 03:47:58,950 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-08-14 03:47:59,080 - DEBUG - Layer2 route found for VM MAC 02:c6:46:68:ad:82 in Control-node 10.0.0.38 2025-08-14 03:47:59,080 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.38 and Agent, Expected: 13, Seen: 13 2025-08-14 03:47:59,080 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-08-14 03:47:59,207 - DEBUG - Layer2 route found for VM MAC 02:c6:46:68:ad:82 in Control-node 10.0.0.27 2025-08-14 03:47:59,208 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.27 and Agent, Expected: 13, Seen: 13 2025-08-14 03:47:59,208 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-08-14 03:47:59,334 - DEBUG - Layer2 route found for VM MAC 02:c6:46:68:ad:82 in Control-node 10.0.0.23 2025-08-14 03:47:59,334 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.23 and Agent, Expected: 13, Seen: 13 2025-08-14 03:47:59,334 - INFO - Verification in Control-nodes for VM EVPN_VN1_VM2 passed 2025-08-14 03:47:59,334 - DEBUG - Verifying the vm in opserver 2025-08-14 03:47:59,334 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-08-14 03:47:59,334 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat 2025-08-14 03:47:59,349 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197?flat 2025-08-14 03:47:59,369 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197?flat 2025-08-14 03:47:59,390 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat 2025-08-14 03:47:59,407 - DEBUG - Verifying vm in vn uve 2025-08-14 03:47:59,408 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197?flat 2025-08-14 03:47:59,428 - DEBUG - VM uve shows interface as {'uuid': 'c64668ad-82ee-4572-b7c2-d4b6c3059197', 'ip6_address': '::', 'active': True, 'ip4_active': True, 'fixed_ip4_list': ['11.1.1.4'], 'ip_address': '11.1.1.4', 'vm_name': 'EVPN_VN1_VM2', 'virtual_network': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'l2_active': True, 'admin_state': True, 'port_mirror_enabled': False, '__T': 1755143261990151, 'label': 25, 'vhostuser_mode': 0, 'tx_vlan': 65535, 'vm_uuid': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'rx_vlan': 65535, 'ip6_active': False, 'vn_uuid': 'a9e32396-5694-4484-8edf-cb76eba7705d', 'mac_address': '02:c6:46:68:ad:82', 'gateway': '11.1.1.1', 'is_health_check_active': True} 2025-08-14 03:47:59,428 - DEBUG - VM uve shows ip address as ['11.1.1.4', '::'] 2025-08-14 03:47:59,428 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:47:59,428 - INFO - Verifying through opserver in 10.0.0.38 2025-08-14 03:47:59,428 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1?flat 2025-08-14 03:47:59,456 - DEBUG - expected vm list 5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:47:59,457 - DEBUG - Extracted vm list ['5f73607a-8d34-4559-942c-31e3300e7fb9'] 2025-08-14 03:47:59,457 - INFO - VM 5f73607a-8d34-4559-942c-31e3300e7fb9 is present in default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:47:59,457 - DEBUG - Verifying vm in vrouter uve 2025-08-14 03:47:59,457 - DEBUG - Getting info from collector 10.0.0.23.. 2025-08-14 03:47:59,457 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat 2025-08-14 03:47:59,472 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-3851-1'] 2025-08-14 03:47:59,472 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-1:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:47:59,514 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-1?flat 2025-08-14 03:47:59,551 - DEBUG - VM 5f73607a-8d34-4559-942c-31e3300e7fb9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-1 uve 2025-08-14 03:47:59,551 - DEBUG - Validated that VM EVPN_VN1_VM2 is in Vrouter an-jenkins-deploy-platform-ansible-os-3851-1 UVE 2025-08-14 03:47:59,551 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-08-14 03:47:59,557 - DEBUG - Expected tap interface of VM uuid 5f73607a-8d34-4559-942c-31e3300e7fb9 is default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197 2025-08-14 03:47:59,558 - DEBUG - Expected VN of VM uuid 5f73607a-8d34-4559-942c-31e3300e7fb9 is default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:47:59,558 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-1:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:47:59,602 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-1?flat 2025-08-14 03:47:59,636 - DEBUG - VM 5f73607a-8d34-4559-942c-31e3300e7fb9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-1 uve 2025-08-14 03:47:59,636 - INFO - tap interface default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197 of vm 5f73607a-8d34-4559-942c-31e3300e7fb9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-1 uve 2025-08-14 03:47:59,636 - DEBUG - Connected network default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 of vm 5f73607a-8d34-4559-942c-31e3300e7fb9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-1 uve 2025-08-14 03:47:59,636 - DEBUG - Verifying the 5f73607a-8d34-4559-942c-31e3300e7fb9 virtual network link through opserver 10.0.0.38 2025-08-14 03:47:59,636 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-14 03:47:59,647 - DEBUG - VM link and name as {'name': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat'} 2025-08-14 03:47:59,647 - DEBUG - Verifying the 5f73607a-8d34-4559-942c-31e3300e7fb9 virtual network link through opserver 10.0.0.27 2025-08-14 03:47:59,647 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-08-14 03:47:59,659 - DEBUG - VM link and name as {'name': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat'} 2025-08-14 03:47:59,659 - DEBUG - Verifying the 5f73607a-8d34-4559-942c-31e3300e7fb9 virtual network link through opserver 10.0.0.23 2025-08-14 03:47:59,659 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-08-14 03:47:59,671 - DEBUG - VM link and name as {'name': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat'} 2025-08-14 03:47:59,671 - INFO - VM EVPN_VN1_VM2 validations in Opserver passed 2025-08-14 03:47:59,671 - INFO - Waiting for VM EVPN_VN1_VM1 to be up.. 2025-08-14 03:47:59,671 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:47:59,671 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:47:59,679 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ee63101f-4639-454c-b775-2ccd7ed65044 2025-08-14 03:48:00,892 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.30 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=65.8 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.299/34.070/65.842/31.771 ms') 2025-08-14 03:48:00,892 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-08-14 03:48:00,965 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:48:00,965 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM1, IP 11.1.1.3, Port 22 2025-08-14 03:48:01,033 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-14 03:48:01,103 - DEBUG - VM EVPN_VN1_VM1 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-14 03:48:06,103 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:48:06,104 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM1, IP 11.1.1.3, Port 22 2025-08-14 03:48:06,158 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-14 03:48:06,232 - DEBUG - VM EVPN_VN1_VM1 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-14 03:48:11,234 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:48:11,234 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM1, IP 11.1.1.3, Port 22 2025-08-14 03:48:11,401 - DEBUG - VM EVPN_VN1_VM1 is ready for SSH connections 2025-08-14 03:48:11,401 - INFO - Waiting for VM EVPN_VN1_VM2 to be up.. 2025-08-14 03:48:11,401 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:48:11,401 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:48:11,411 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c64668ad-82ee-4572-b7c2-d4b6c3059197 2025-08-14 03:48:12,579 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.90 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=5.18 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 3.895/4.538/5.182/0.643 ms') 2025-08-14 03:48:12,579 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM2 passed 2025-08-14 03:48:12,644 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:48:12,644 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM2, IP 11.1.1.4, Port 22 2025-08-14 03:48:12,712 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-14 03:48:12,781 - DEBUG - VM EVPN_VN1_VM2 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-14 03:48:17,781 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:48:17,781 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM2, IP 11.1.1.4, Port 22 2025-08-14 03:48:17,952 - DEBUG - VM EVPN_VN1_VM2 is ready for SSH connections 2025-08-14 03:48:18,021 - DEBUG - Executing command: sudo tcpdump -nni tapc64668ad-82 -U ether src 02:ee:63:10:1f:46 -w /tmp/tapc64668ad-82_ctest-random-56585444.pcap 2025-08-14 03:48:18,389 - INFO - waiting to get tcpdump started 2025-08-14 03:48:41,394 - DEBUG - file copy failed retrying 2025-08-14 03:48:44,396 - DEBUG - file copy failed retrying 2025-08-14 03:49:17,226 - DEBUG - file transfer done ['/tmp/program.py'] 2025-08-14 03:49:17,226 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:49:17,226 - DEBUG - Copied file /tmp/tmp77fjfnk2/program.py to 11.1.1.3 2025-08-14 03:49:17,226 - DEBUG - Running remote_cmd, Cmd : python /tmp/program.py, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-08-14 03:49:17,227 - DEBUG - python /tmp/program.py 2025-08-14 03:49:19,301 - DEBUG - WARNING: No route found for IPv6 destination :: (no default route?) .......... Sent 10 packets. 2025-08-14 03:49:19,302 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-56585444.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:49:19,315 - DEBUG - STDOUT: 0 2025-08-14 03:49:19,315 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-56585444.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:49:19,315 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-56585444.pcap but expected 10 2025-08-14 03:49:21,316 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-56585444.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:49:21,329 - DEBUG - STDOUT: 0 2025-08-14 03:49:21,329 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-56585444.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:49:21,329 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-56585444.pcap but expected 10 2025-08-14 03:49:23,329 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-56585444.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:49:23,342 - DEBUG - STDOUT: 0 2025-08-14 03:49:23,342 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-56585444.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:49:23,342 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-56585444.pcap but expected 10 2025-08-14 03:49:25,342 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-56585444.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:49:25,357 - DEBUG - STDOUT: 0 2025-08-14 03:49:25,357 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-56585444.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:49:25,357 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-56585444.pcap but expected 10 2025-08-14 03:49:27,359 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-56585444.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:49:27,371 - DEBUG - STDOUT: 0 2025-08-14 03:49:27,371 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-56585444.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:49:27,372 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-56585444.pcap but expected 10 2025-08-14 03:49:29,373 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-56585444.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:49:29,386 - DEBUG - STDOUT: 0 2025-08-14 03:49:29,386 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-56585444.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:49:29,386 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-56585444.pcap but expected 10 2025-08-14 03:49:31,386 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-56585444.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:49:31,400 - DEBUG - STDOUT: 0 2025-08-14 03:49:31,400 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-56585444.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:49:31,400 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-56585444.pcap but expected 10 2025-08-14 03:49:31,400 - INFO - Will restart compute services now 2025-08-14 03:49:35,168 - INFO - 10.0.0.65 2025-08-14 03:49:35,173 - INFO - agent:initializing 2025-08-14 03:49:35,173 - DEBUG - defaultdict(, {'10.0.0.65': {'agent': {'status': 'initializing', 'description': None}}}) 2025-08-14 03:49:35,173 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-14 03:49:40,264 - INFO - 10.0.0.65 2025-08-14 03:49:40,270 - INFO - agent:active 2025-08-14 03:49:40,270 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.65 2025-08-14 03:49:43,630 - INFO - 10.0.0.130 2025-08-14 03:49:43,634 - INFO - agent:initializing 2025-08-14 03:49:43,635 - DEBUG - defaultdict(, {'10.0.0.130': {'agent': {'status': 'initializing', 'description': None}}}) 2025-08-14 03:49:43,635 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-14 03:49:48,726 - INFO - 10.0.0.130 2025-08-14 03:49:48,733 - INFO - agent:active 2025-08-14 03:49:48,733 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.130 2025-08-14 03:49:58,734 - INFO - Verifying L2 route and other VM verification after restart 2025-08-14 03:49:58,817 - DEBUG - VM is in ACTIVE state now 2025-08-14 03:49:58,817 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:49:58,817 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:49:58,828 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ee63101f-4639-454c-b775-2ccd7ed65044 2025-08-14 03:49:58,839 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:49:58,849 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ee63101f-4639-454c-b775-2ccd7ed65044 2025-08-14 03:49:58,860 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/cbe64ad8-5d88-418b-a864-7436bb85523a 2025-08-14 03:49:58,869 - DEBUG - Verifying in api server 10.0.0.23 2025-08-14 03:49:58,869 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:49:58,877 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/ee63101f-4639-454c-b775-2ccd7ed65044 2025-08-14 03:49:58,888 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/cbe64ad8-5d88-418b-a864-7436bb85523a 2025-08-14 03:49:58,897 - INFO - VM EVPN_VN1_VM1 verfication in all API Servers passed 2025-08-14 03:49:58,915 - DEBUG - VM EVPN_VN1_VM1 Tap interface: {'index': '3', 'name': 'tapee63101f-46', 'uuid': 'ee63101f-4639-454c-b775-2ccd7ed65044', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '23', 'l2_label': '27', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'vm_uuid': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'vm_name': 'EVPN_VN1_VM1', 'ip_addr': '11.1.1.3', 'mac_addr': '02:ee:63:10:1f:46', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '150', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044', 'sg_uuid_list': ['0acf68aa-9333-4001-a6ce-69695242eb74'], 'static_route_list': None, 'vm_project_uuid': '267caa7c-ab72-42d1-a420-7d6ef32f5c83', 'admin_state': 'Enabled', 'flow_key_idx': '23', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-14 03:49:58,915 - DEBUG - Agent 10.0.0.130 vrf name: default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1 2025-08-14 03:49:58,921 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-08-14 03:49:59,006 - DEBUG - VM EVPN_VN1_VM1 : Tap interface tapee63101f-46 is set to Active 2025-08-14 03:49:59,006 - DEBUG - Tap interface tapee63101f-46 detail : {'index': '3', 'name': 'tapee63101f-46', 'uuid': 'ee63101f-4639-454c-b775-2ccd7ed65044', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '23', 'l2_label': '27', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'vm_uuid': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'vm_name': 'EVPN_VN1_VM1', 'ip_addr': '11.1.1.3', 'mac_addr': '02:ee:63:10:1f:46', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '150', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044', 'sg_uuid_list': ['0acf68aa-9333-4001-a6ce-69695242eb74'], 'static_route_list': None, 'vm_project_uuid': '267caa7c-ab72-42d1-a420-7d6ef32f5c83', 'admin_state': 'Enabled', 'flow_key_idx': '23', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-14 03:49:59,122 - DEBUG - Starting Layer 2 verification in Agent 2025-08-14 03:49:59,133 - DEBUG - Layer 2 path is seen for VM MAC 02:ee:63:10:1f:46 in agent 10.0.0.130 2025-08-14 03:49:59,133 - DEBUG - Active layer 2 route in agent is present for VMI tapee63101f-46 2025-08-14 03:49:59,133 - DEBUG - vxlan_id (13) matches bw route table and interface table 2025-08-14 03:49:59,139 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is found in Agent of node 10.0.0.65 2025-08-14 03:49:59,273 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is consistent in agent 10.0.0.65 2025-08-14 03:49:59,273 - DEBUG - Route for VM IP ['11.1.1.3'] is consistent in agent 10.0.0.65 2025-08-14 03:49:59,273 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 verification for VM EVPN_VN1_VM1 in Agent 10.0.0.65 passed 2025-08-14 03:49:59,394 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2025-08-14 03:49:59,405 - DEBUG - Route for VM MAC 02:ee:63:10:1f:46 is consistent in agent 10.0.0.65 2025-08-14 03:49:59,412 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is found in Agent of node 10.0.0.130 2025-08-14 03:49:59,552 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is consistent in agent 10.0.0.130 2025-08-14 03:49:59,553 - DEBUG - Route for VM IP ['11.1.1.3'] is consistent in agent 10.0.0.130 2025-08-14 03:49:59,553 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 verification for VM EVPN_VN1_VM1 in Agent 10.0.0.130 passed 2025-08-14 03:49:59,669 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-08-14 03:49:59,681 - DEBUG - Route for VM MAC 02:ee:63:10:1f:46 is consistent in agent 10.0.0.130 2025-08-14 03:50:00,865 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.15 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.770 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.770/1.462/2.154/0.692 ms') 2025-08-14 03:50:00,865 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-08-14 03:50:00,865 - INFO - VM EVPN_VN1_VM1 verifications in Compute nodes passed 2025-08-14 03:50:01,057 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.65 2025-08-14 03:50:01,059 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.192, 23; Got 10.20.0.192,23 2025-08-14 03:50:01,060 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.65 2025-08-14 03:50:01,170 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.130 2025-08-14 03:50:01,174 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-08-14 03:50:01,174 - INFO - Validated routes of VM EVPN_VN1_VM1 in all vrouters 2025-08-14 03:50:01,687 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-08-14 03:50:01,810 - DEBUG - Layer2 route found for VM MAC 02:ee:63:10:1f:46 in Control-node 10.0.0.38 2025-08-14 03:50:01,810 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.38 and Agent, Expected: 13, Seen: 13 2025-08-14 03:50:01,810 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-08-14 03:50:01,938 - DEBUG - Layer2 route found for VM MAC 02:ee:63:10:1f:46 in Control-node 10.0.0.27 2025-08-14 03:50:01,938 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.27 and Agent, Expected: 13, Seen: 13 2025-08-14 03:50:01,938 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-08-14 03:50:02,062 - DEBUG - Layer2 route found for VM MAC 02:ee:63:10:1f:46 in Control-node 10.0.0.23 2025-08-14 03:50:02,062 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.23 and Agent, Expected: 13, Seen: 13 2025-08-14 03:50:02,062 - INFO - Verification in Control-nodes for VM EVPN_VN1_VM1 passed 2025-08-14 03:50:02,062 - DEBUG - Verifying the vm in opserver 2025-08-14 03:50:02,062 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-08-14 03:50:02,062 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:50:02,077 - DEBUG - Failed to get VM EVPN_VN1_VM1, ID f40f0938-5b54-4c7c-8ac7-14b360cbabc9 info from Opserver 2025-08-14 03:50:04,078 - DEBUG - Verifying the vm in opserver 2025-08-14 03:50:04,078 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-08-14 03:50:04,079 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:50:04,094 - DEBUG - Failed to get VM EVPN_VN1_VM1, ID f40f0938-5b54-4c7c-8ac7-14b360cbabc9 info from Opserver 2025-08-14 03:50:06,095 - DEBUG - Verifying the vm in opserver 2025-08-14 03:50:06,095 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-08-14 03:50:06,095 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:50:06,111 - DEBUG - Failed to get VM EVPN_VN1_VM1, ID f40f0938-5b54-4c7c-8ac7-14b360cbabc9 info from Opserver 2025-08-14 03:50:08,111 - DEBUG - Verifying the vm in opserver 2025-08-14 03:50:08,111 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-08-14 03:50:08,111 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:50:08,130 - DEBUG - Failed to get VM EVPN_VN1_VM1, ID f40f0938-5b54-4c7c-8ac7-14b360cbabc9 info from Opserver 2025-08-14 03:50:10,131 - DEBUG - Verifying the vm in opserver 2025-08-14 03:50:10,131 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-08-14 03:50:10,131 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:50:10,149 - DEBUG - Failed to get VM EVPN_VN1_VM1, ID f40f0938-5b54-4c7c-8ac7-14b360cbabc9 info from Opserver 2025-08-14 03:50:12,149 - DEBUG - Verifying the vm in opserver 2025-08-14 03:50:12,149 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-08-14 03:50:12,149 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:50:12,166 - DEBUG - Failed to get VM EVPN_VN1_VM1, ID f40f0938-5b54-4c7c-8ac7-14b360cbabc9 info from Opserver 2025-08-14 03:50:14,166 - DEBUG - Verifying the vm in opserver 2025-08-14 03:50:14,166 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-08-14 03:50:14,167 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:50:14,186 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044?flat 2025-08-14 03:50:14,208 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044?flat 2025-08-14 03:50:14,230 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:50:14,250 - DEBUG - Verifying vm in vn uve 2025-08-14 03:50:14,250 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044?flat 2025-08-14 03:50:14,271 - DEBUG - VM uve shows interface as {'uuid': 'ee63101f-4639-454c-b775-2ccd7ed65044', 'ip6_address': '::', 'active': True, 'ip4_active': True, 'fixed_ip4_list': ['11.1.1.3'], 'ip_address': '11.1.1.3', 'vm_name': 'EVPN_VN1_VM1', 'virtual_network': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'l2_active': True, 'admin_state': True, 'port_mirror_enabled': False, '__T': 1755143414123066, 'label': 23, 'vhostuser_mode': 0, 'tx_vlan': 65535, 'vm_uuid': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'rx_vlan': 65535, 'ip6_active': False, 'vn_uuid': 'a9e32396-5694-4484-8edf-cb76eba7705d', 'mac_address': '02:ee:63:10:1f:46', 'gateway': '11.1.1.1', 'is_health_check_active': True} 2025-08-14 03:50:14,271 - DEBUG - VM uve shows ip address as ['11.1.1.3', '::'] 2025-08-14 03:50:14,271 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:50:14,271 - INFO - Verifying through opserver in 10.0.0.38 2025-08-14 03:50:14,271 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1?flat 2025-08-14 03:50:14,302 - DEBUG - expected vm list f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:50:14,302 - DEBUG - Extracted vm list ['f40f0938-5b54-4c7c-8ac7-14b360cbabc9'] 2025-08-14 03:50:14,302 - INFO - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is present in default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:50:14,302 - DEBUG - Verifying vm in vrouter uve 2025-08-14 03:50:14,302 - DEBUG - Getting info from collector 10.0.0.23.. 2025-08-14 03:50:14,302 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat 2025-08-14 03:50:14,322 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-3851-2'] 2025-08-14 03:50:14,322 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-2:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:50:14,367 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-2?flat 2025-08-14 03:50:14,402 - DEBUG - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:50:14,402 - DEBUG - Validated that VM EVPN_VN1_VM1 is in Vrouter an-jenkins-deploy-platform-ansible-os-3851-2 UVE 2025-08-14 03:50:14,402 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-08-14 03:50:14,410 - DEBUG - Expected tap interface of VM uuid f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044 2025-08-14 03:50:14,410 - DEBUG - Expected VN of VM uuid f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:50:14,410 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-2:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:50:14,521 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-2?flat 2025-08-14 03:50:14,556 - DEBUG - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:50:14,556 - INFO - tap interface default-domain:ctest-TestEvpnCasesRestart-59825578:ee63101f-4639-454c-b775-2ccd7ed65044 of vm f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:50:14,556 - DEBUG - Connected network default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 of vm f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:50:14,556 - DEBUG - Verifying the f40f0938-5b54-4c7c-8ac7-14b360cbabc9 virtual network link through opserver 10.0.0.38 2025-08-14 03:50:14,556 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-14 03:50:14,569 - DEBUG - VM link and name as {'name': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat'} 2025-08-14 03:50:14,569 - DEBUG - Verifying the f40f0938-5b54-4c7c-8ac7-14b360cbabc9 virtual network link through opserver 10.0.0.27 2025-08-14 03:50:14,569 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-08-14 03:50:14,584 - DEBUG - VM link and name as {'name': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat'} 2025-08-14 03:50:14,584 - DEBUG - Verifying the f40f0938-5b54-4c7c-8ac7-14b360cbabc9 virtual network link through opserver 10.0.0.23 2025-08-14 03:50:14,584 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-08-14 03:50:14,596 - DEBUG - VM link and name as {'name': 'f40f0938-5b54-4c7c-8ac7-14b360cbabc9', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9?flat'} 2025-08-14 03:50:14,596 - INFO - VM EVPN_VN1_VM1 validations in Opserver passed 2025-08-14 03:50:14,667 - DEBUG - VM is in ACTIVE state now 2025-08-14 03:50:14,667 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:50:14,667 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:50:14,676 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c64668ad-82ee-4572-b7c2-d4b6c3059197 2025-08-14 03:50:14,687 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:50:14,694 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c64668ad-82ee-4572-b7c2-d4b6c3059197 2025-08-14 03:50:14,704 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/37b80726-66bd-4187-a2e2-cc3e526d8182 2025-08-14 03:50:14,713 - DEBUG - Verifying in api server 10.0.0.23 2025-08-14 03:50:14,713 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:50:14,722 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c64668ad-82ee-4572-b7c2-d4b6c3059197 2025-08-14 03:50:14,733 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/37b80726-66bd-4187-a2e2-cc3e526d8182 2025-08-14 03:50:14,741 - INFO - VM EVPN_VN1_VM2 verfication in all API Servers passed 2025-08-14 03:50:14,761 - DEBUG - VM EVPN_VN1_VM2 Tap interface: {'index': '3', 'name': 'tapc64668ad-82', 'uuid': 'c64668ad-82ee-4572-b7c2-d4b6c3059197', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '23', 'l2_label': '27', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'vm_uuid': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'vm_name': 'EVPN_VN1_VM2', 'ip_addr': '11.1.1.4', 'mac_addr': '02:c6:46:68:ad:82', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '112', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197', 'sg_uuid_list': ['0acf68aa-9333-4001-a6ce-69695242eb74'], 'static_route_list': None, 'vm_project_uuid': '267caa7c-ab72-42d1-a420-7d6ef32f5c83', 'admin_state': 'Enabled', 'flow_key_idx': '20', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-14 03:50:14,761 - DEBUG - Agent 10.0.0.65 vrf name: default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1 2025-08-14 03:50:14,769 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.18:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-08-14 03:50:14,851 - DEBUG - VM EVPN_VN1_VM2 : Tap interface tapc64668ad-82 is set to Active 2025-08-14 03:50:14,851 - DEBUG - Tap interface tapc64668ad-82 detail : {'index': '3', 'name': 'tapc64668ad-82', 'uuid': 'c64668ad-82ee-4572-b7c2-d4b6c3059197', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '23', 'l2_label': '27', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'vm_uuid': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'vm_name': 'EVPN_VN1_VM2', 'ip_addr': '11.1.1.4', 'mac_addr': '02:c6:46:68:ad:82', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '112', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197', 'sg_uuid_list': ['0acf68aa-9333-4001-a6ce-69695242eb74'], 'static_route_list': None, 'vm_project_uuid': '267caa7c-ab72-42d1-a420-7d6ef32f5c83', 'admin_state': 'Enabled', 'flow_key_idx': '20', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-14 03:50:14,964 - DEBUG - Starting Layer 2 verification in Agent 2025-08-14 03:50:14,979 - DEBUG - Layer 2 path is seen for VM MAC 02:c6:46:68:ad:82 in agent 10.0.0.65 2025-08-14 03:50:14,979 - DEBUG - Active layer 2 route in agent is present for VMI tapc64668ad-82 2025-08-14 03:50:14,979 - DEBUG - vxlan_id (13) matches bw route table and interface table 2025-08-14 03:50:14,988 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is found in Agent of node 10.0.0.65 2025-08-14 03:50:15,124 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is consistent in agent 10.0.0.65 2025-08-14 03:50:15,124 - DEBUG - Route for VM IP ['11.1.1.4'] is consistent in agent 10.0.0.65 2025-08-14 03:50:15,124 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 verification for VM EVPN_VN1_VM2 in Agent 10.0.0.65 passed 2025-08-14 03:50:15,237 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2025-08-14 03:50:15,248 - DEBUG - Route for VM MAC 02:c6:46:68:ad:82 is consistent in agent 10.0.0.65 2025-08-14 03:50:15,254 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is found in Agent of node 10.0.0.130 2025-08-14 03:50:15,389 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 is consistent in agent 10.0.0.130 2025-08-14 03:50:15,390 - DEBUG - Route for VM IP ['11.1.1.4'] is consistent in agent 10.0.0.130 2025-08-14 03:50:15,390 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 verification for VM EVPN_VN1_VM2 in Agent 10.0.0.130 passed 2025-08-14 03:50:15,505 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-08-14 03:50:15,516 - DEBUG - Route for VM MAC 02:c6:46:68:ad:82 is consistent in agent 10.0.0.130 2025-08-14 03:50:16,699 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.12 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.511 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.511/1.313/2.116/0.802 ms') 2025-08-14 03:50:16,699 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM2 passed 2025-08-14 03:50:16,699 - INFO - VM EVPN_VN1_VM2 verifications in Compute nodes passed 2025-08-14 03:50:16,895 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.65 2025-08-14 03:50:16,898 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-08-14 03:50:17,013 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.130 2025-08-14 03:50:17,017 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.18, 23; Got 10.20.0.18,23 2025-08-14 03:50:17,017 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.130 2025-08-14 03:50:17,017 - INFO - Validated routes of VM EVPN_VN1_VM2 in all vrouters 2025-08-14 03:50:17,502 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-08-14 03:50:17,626 - DEBUG - Layer2 route found for VM MAC 02:c6:46:68:ad:82 in Control-node 10.0.0.38 2025-08-14 03:50:17,626 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.38 and Agent, Expected: 13, Seen: 13 2025-08-14 03:50:17,626 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-08-14 03:50:17,757 - DEBUG - Layer2 route found for VM MAC 02:c6:46:68:ad:82 in Control-node 10.0.0.27 2025-08-14 03:50:17,757 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.27 and Agent, Expected: 13, Seen: 13 2025-08-14 03:50:17,758 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-08-14 03:50:17,886 - DEBUG - Layer2 route found for VM MAC 02:c6:46:68:ad:82 in Control-node 10.0.0.23 2025-08-14 03:50:17,886 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.23 and Agent, Expected: 13, Seen: 13 2025-08-14 03:50:17,886 - INFO - Verification in Control-nodes for VM EVPN_VN1_VM2 passed 2025-08-14 03:50:17,886 - DEBUG - Verifying the vm in opserver 2025-08-14 03:50:17,886 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-08-14 03:50:17,886 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat 2025-08-14 03:50:17,902 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197?flat 2025-08-14 03:50:17,923 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197?flat 2025-08-14 03:50:17,943 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat 2025-08-14 03:50:17,959 - DEBUG - Verifying vm in vn uve 2025-08-14 03:50:17,959 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197?flat 2025-08-14 03:50:17,978 - DEBUG - VM uve shows interface as {'uuid': 'c64668ad-82ee-4572-b7c2-d4b6c3059197', 'ip6_address': '::', 'active': True, 'ip4_active': True, 'fixed_ip4_list': ['11.1.1.4'], 'ip_address': '11.1.1.4', 'vm_name': 'EVPN_VN1_VM2', 'virtual_network': 'default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1', 'l2_active': True, 'admin_state': True, 'port_mirror_enabled': False, '__T': 1755143405717096, 'label': 23, 'vhostuser_mode': 0, 'tx_vlan': 65535, 'vm_uuid': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'rx_vlan': 65535, 'ip6_active': False, 'vn_uuid': 'a9e32396-5694-4484-8edf-cb76eba7705d', 'mac_address': '02:c6:46:68:ad:82', 'gateway': '11.1.1.1', 'is_health_check_active': True} 2025-08-14 03:50:17,978 - DEBUG - VM uve shows ip address as ['11.1.1.4', '::'] 2025-08-14 03:50:17,978 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:50:17,978 - INFO - Verifying through opserver in 10.0.0.38 2025-08-14 03:50:17,978 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1?flat 2025-08-14 03:50:18,008 - DEBUG - expected vm list 5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:50:18,008 - DEBUG - Extracted vm list ['5f73607a-8d34-4559-942c-31e3300e7fb9'] 2025-08-14 03:50:18,008 - INFO - VM 5f73607a-8d34-4559-942c-31e3300e7fb9 is present in default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:50:18,008 - DEBUG - Verifying vm in vrouter uve 2025-08-14 03:50:18,008 - DEBUG - Getting info from collector 10.0.0.23.. 2025-08-14 03:50:18,008 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat 2025-08-14 03:50:18,025 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-3851-1'] 2025-08-14 03:50:18,025 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-1:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:50:18,066 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-1?flat 2025-08-14 03:50:18,104 - DEBUG - VM 5f73607a-8d34-4559-942c-31e3300e7fb9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-1 uve 2025-08-14 03:50:18,104 - DEBUG - Validated that VM EVPN_VN1_VM2 is in Vrouter an-jenkins-deploy-platform-ansible-os-3851-1 UVE 2025-08-14 03:50:18,104 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-08-14 03:50:18,113 - DEBUG - Expected tap interface of VM uuid 5f73607a-8d34-4559-942c-31e3300e7fb9 is default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197 2025-08-14 03:50:18,113 - DEBUG - Expected VN of VM uuid 5f73607a-8d34-4559-942c-31e3300e7fb9 is default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:50:18,113 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-1:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:50:18,173 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-1?flat 2025-08-14 03:50:18,211 - DEBUG - VM 5f73607a-8d34-4559-942c-31e3300e7fb9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-1 uve 2025-08-14 03:50:18,211 - INFO - tap interface default-domain:ctest-TestEvpnCasesRestart-59825578:c64668ad-82ee-4572-b7c2-d4b6c3059197 of vm 5f73607a-8d34-4559-942c-31e3300e7fb9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-1 uve 2025-08-14 03:50:18,212 - DEBUG - Connected network default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 of vm 5f73607a-8d34-4559-942c-31e3300e7fb9 is present in vrouter an-jenkins-deploy-platform-ansible-os-3851-1 uve 2025-08-14 03:50:18,212 - DEBUG - Verifying the 5f73607a-8d34-4559-942c-31e3300e7fb9 virtual network link through opserver 10.0.0.38 2025-08-14 03:50:18,212 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-14 03:50:18,224 - DEBUG - VM link and name as {'name': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat'} 2025-08-14 03:50:18,225 - DEBUG - Verifying the 5f73607a-8d34-4559-942c-31e3300e7fb9 virtual network link through opserver 10.0.0.27 2025-08-14 03:50:18,225 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-08-14 03:50:18,239 - DEBUG - VM link and name as {'name': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat'} 2025-08-14 03:50:18,239 - DEBUG - Verifying the 5f73607a-8d34-4559-942c-31e3300e7fb9 virtual network link through opserver 10.0.0.23 2025-08-14 03:50:18,239 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-08-14 03:50:18,253 - DEBUG - VM link and name as {'name': '5f73607a-8d34-4559-942c-31e3300e7fb9', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9?flat'} 2025-08-14 03:50:18,253 - INFO - VM EVPN_VN1_VM2 validations in Opserver passed 2025-08-14 03:50:18,322 - DEBUG - Executing command: sudo tcpdump -nni tapc64668ad-82 -U ether src 02:ee:63:10:1f:46 -w /tmp/tapc64668ad-82_ctest-random-70031251.pcap 2025-08-14 03:50:18,409 - INFO - waiting to get tcpdump started 2025-08-14 03:50:39,550 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=0.754 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.605 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1012ms\r\nrtt min/avg/max/mdev = 0.605/0.679/0.754/0.074 ms') 2025-08-14 03:50:39,550 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-08-14 03:50:39,702 - DEBUG - file transfer done ['/tmp/program.py'] 2025-08-14 03:50:39,702 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-14 03:50:39,702 - DEBUG - Copied file /tmp/tmphadr14a2/program.py to 11.1.1.3 2025-08-14 03:50:40,831 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=1.40 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.629 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.629/1.013/1.397/0.384 ms') 2025-08-14 03:50:40,831 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-08-14 03:50:40,831 - DEBUG - Running remote_cmd, Cmd : python /tmp/program.py, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-08-14 03:50:40,831 - DEBUG - python /tmp/program.py 2025-08-14 03:50:42,531 - DEBUG - WARNING: No route found for IPv6 destination :: (no default route?) .......... Sent 10 packets. 2025-08-14 03:50:42,532 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-70031251.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:50:42,545 - DEBUG - STDOUT: 0 2025-08-14 03:50:42,545 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-70031251.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:50:42,545 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-70031251.pcap but expected 10 2025-08-14 03:50:44,546 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-70031251.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:50:44,558 - DEBUG - STDOUT: 0 2025-08-14 03:50:44,558 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-70031251.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:50:44,558 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-70031251.pcap but expected 10 2025-08-14 03:50:46,559 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-70031251.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:50:46,572 - DEBUG - STDOUT: 0 2025-08-14 03:50:46,572 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-70031251.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:50:46,572 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-70031251.pcap but expected 10 2025-08-14 03:50:48,572 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-70031251.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:50:48,585 - DEBUG - STDOUT: 0 2025-08-14 03:50:48,585 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-70031251.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:50:48,585 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-70031251.pcap but expected 10 2025-08-14 03:50:50,586 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-70031251.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:50:50,598 - DEBUG - STDOUT: 0 2025-08-14 03:50:50,598 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-70031251.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:50:50,598 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-70031251.pcap but expected 10 2025-08-14 03:50:52,600 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-70031251.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:50:52,613 - DEBUG - STDOUT: 0 2025-08-14 03:50:52,613 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-70031251.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:50:52,613 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-70031251.pcap but expected 10 2025-08-14 03:50:54,614 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapc64668ad-82_ctest-random-70031251.pcap ether host 02:c6:46:68:ad:82 | grep -c length 2025-08-14 03:50:54,627 - DEBUG - STDOUT: 0 2025-08-14 03:50:54,627 - DEBUG - STDERR: reading from file /tmp/tapc64668ad-82_ctest-random-70031251.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-14 03:50:54,627 - WARNING - 0 packets are found in tcpdump output file /tmp/tapc64668ad-82_ctest-random-70031251.pcap but expected 10 2025-08-14 03:50:54,627 - INFO - Checking the communication between 2 VM after vrouter restart 2025-08-14 03:50:54,627 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:50:54,635 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/c64668ad-82ee-4572-b7c2-d4b6c3059197 2025-08-14 03:50:54,645 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/37b80726-66bd-4187-a2e2-cc3e526d8182 2025-08-14 03:50:55,810 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=0.872 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.426 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1031ms\r\nrtt min/avg/max/mdev = 0.426/0.649/0.872/0.223 ms') 2025-08-14 03:50:55,810 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-08-14 03:50:55,810 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 11.1.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-08-14 03:50:55,810 - DEBUG - ping -s 56 -c 3 -W 1 11.1.1.4 2025-08-14 03:50:58,258 - DEBUG - PING 11.1.1.4 (11.1.1.4) 56(84) bytes of data. 64 bytes from 11.1.1.4: icmp_req=1 ttl=64 time=8.85 ms 64 bytes from 11.1.1.4: icmp_req=2 ttl=64 time=1.13 ms 64 bytes from 11.1.1.4: icmp_req=3 ttl=64 time=1.04 ms --- 11.1.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.046/3.679/8.853/3.658 ms 2025-08-14 03:50:58,258 - INFO - Ping to IP 11.1.1.4 from VM EVPN_VN1_VM1 passed 2025-08-14 03:50:58,259 - INFO - Deleting VM EVPN_VN1_VM2 2025-08-14 03:50:58,317 - DEBUG - Verifying in api server 10.0.0.23 2025-08-14 03:50:58,317 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:50:58,326 - DEBUG - VM ID 5f73607a-8d34-4559-942c-31e3300e7fb9 of VM EVPN_VN1_VM2 is still found in API Server 2025-08-14 03:51:00,327 - DEBUG - Verifying in api server 10.0.0.23 2025-08-14 03:51:00,327 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:00,345 - DEBUG - Response Code: 404 2025-08-14 03:51:00,346 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:00,352 - DEBUG - Response Code: 404 2025-08-14 03:51:00,352 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:00,358 - DEBUG - Response Code: 404 2025-08-14 03:51:00,358 - INFO - VM EVPN_VN1_VM2 is fully removed in API-Server 2025-08-14 03:51:01,098 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-08-14 03:51:01,110 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:01,116 - DEBUG - Response Code: 404 2025-08-14 03:51:02,117 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:02,125 - DEBUG - Response Code: 404 2025-08-14 03:51:03,126 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:03,132 - DEBUG - Response Code: 404 2025-08-14 03:51:04,133 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:04,140 - DEBUG - Response Code: 404 2025-08-14 03:51:05,141 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:05,148 - DEBUG - Response Code: 404 2025-08-14 03:51:06,149 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:06,156 - DEBUG - Response Code: 404 2025-08-14 03:51:06,156 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:06,162 - DEBUG - Response Code: 404 2025-08-14 03:51:07,163 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:07,170 - DEBUG - Response Code: 404 2025-08-14 03:51:08,171 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:08,178 - DEBUG - Response Code: 404 2025-08-14 03:51:09,179 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:09,185 - DEBUG - Response Code: 404 2025-08-14 03:51:10,186 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:10,193 - DEBUG - Response Code: 404 2025-08-14 03:51:11,193 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:11,201 - DEBUG - Response Code: 404 2025-08-14 03:51:11,212 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:51:11,222 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:51:11,222 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:51:11,222 - INFO - VM EVPN_VN1_VM2 is removed in Compute, and routes are removed in all compute nodes 2025-08-14 03:51:11,458 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:11,464 - DEBUG - Response Code: 404 2025-08-14 03:51:12,465 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:12,472 - DEBUG - Response Code: 404 2025-08-14 03:51:13,472 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:13,479 - DEBUG - Response Code: 404 2025-08-14 03:51:14,480 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:14,487 - DEBUG - Response Code: 404 2025-08-14 03:51:15,488 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:15,496 - DEBUG - Response Code: 404 2025-08-14 03:51:16,496 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:16,504 - DEBUG - Response Code: 404 2025-08-14 03:51:16,504 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:16,510 - DEBUG - Response Code: 404 2025-08-14 03:51:17,510 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:17,518 - DEBUG - Response Code: 404 2025-08-14 03:51:18,519 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:18,530 - DEBUG - Response Code: 404 2025-08-14 03:51:19,531 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:19,543 - DEBUG - Response Code: 404 2025-08-14 03:51:20,545 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:20,552 - DEBUG - Response Code: 404 2025-08-14 03:51:21,552 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/5f73607a-8d34-4559-942c-31e3300e7fb9 2025-08-14 03:51:21,563 - DEBUG - Response Code: 404 2025-08-14 03:51:21,563 - INFO - Routes for VM EVPN_VN1_VM2 is removed in all control-nodes 2025-08-14 03:51:21,563 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-1:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:51:21,608 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-1?flat 2025-08-14 03:51:21,645 - DEBUG - VM 5f73607a-8d34-4559-942c-31e3300e7fb9 is not present in vrouter an-jenkins-deploy-platform-ansible-os-3851-1 uve 2025-08-14 03:51:21,645 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-14 03:51:21,645 - DEBUG - interface for vm 5f73607a-8d34-4559-942c-31e3300e7fb9 deleted from vrouter uve ... 2025-08-14 03:51:21,645 - DEBUG - Deleted interface not in error interface list ... 2025-08-14 03:51:21,645 - DEBUG - Validated that VM 5f73607a-8d34-4559-942c-31e3300e7fb9 is removed in Vrouter UVE 2025-08-14 03:51:21,645 - DEBUG - Verifying through opserver in 10.0.0.38 2025-08-14 03:51:21,646 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1?flat 2025-08-14 03:51:21,674 - DEBUG - 5f73607a-8d34-4559-942c-31e3300e7fb9 BM not in default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 uve 2025-08-14 03:51:21,674 - DEBUG - Validated that VM 5f73607a-8d34-4559-942c-31e3300e7fb9 is not present in the VN UVE 2025-08-14 03:51:21,674 - DEBUG - Verifying the 5f73607a-8d34-4559-942c-31e3300e7fb9 virtual network link through opserver 10.0.0.38 2025-08-14 03:51:21,674 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-14 03:51:21,687 - INFO - 5f73607a-8d34-4559-942c-31e3300e7fb9 vm uve deleted from opserver 2025-08-14 03:51:21,687 - DEBUG - Verifying the 5f73607a-8d34-4559-942c-31e3300e7fb9 virtual network link through opserver 10.0.0.27 2025-08-14 03:51:21,687 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-08-14 03:51:21,701 - INFO - 5f73607a-8d34-4559-942c-31e3300e7fb9 vm uve deleted from opserver 2025-08-14 03:51:21,702 - DEBUG - Verifying the 5f73607a-8d34-4559-942c-31e3300e7fb9 virtual network link through opserver 10.0.0.23 2025-08-14 03:51:21,702 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-08-14 03:51:21,717 - INFO - 5f73607a-8d34-4559-942c-31e3300e7fb9 vm uve deleted from opserver 2025-08-14 03:51:21,717 - INFO - Validated that VM 5f73607a-8d34-4559-942c-31e3300e7fb9 is removed from Opserver 2025-08-14 03:51:21,717 - INFO - Deleting VM EVPN_VN1_VM1 2025-08-14 03:51:21,776 - DEBUG - Verifying in api server 10.0.0.23 2025-08-14 03:51:21,776 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:21,785 - DEBUG - VM ID f40f0938-5b54-4c7c-8ac7-14b360cbabc9 of VM EVPN_VN1_VM1 is still found in API Server 2025-08-14 03:51:23,785 - DEBUG - Verifying in api server 10.0.0.23 2025-08-14 03:51:23,785 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:23,792 - DEBUG - Response Code: 404 2025-08-14 03:51:23,793 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:23,800 - DEBUG - Response Code: 404 2025-08-14 03:51:23,800 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:23,806 - DEBUG - Response Code: 404 2025-08-14 03:51:23,806 - INFO - VM EVPN_VN1_VM1 is fully removed in API-Server 2025-08-14 03:51:23,959 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-08-14 03:51:23,973 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:23,981 - DEBUG - Response Code: 404 2025-08-14 03:51:24,981 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:24,991 - DEBUG - Response Code: 404 2025-08-14 03:51:25,992 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:26,000 - DEBUG - Response Code: 404 2025-08-14 03:51:27,001 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:27,007 - DEBUG - Response Code: 404 2025-08-14 03:51:28,008 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:28,016 - DEBUG - Response Code: 404 2025-08-14 03:51:29,017 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:29,024 - DEBUG - Response Code: 404 2025-08-14 03:51:29,025 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:29,034 - DEBUG - Response Code: 404 2025-08-14 03:51:30,035 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:30,042 - DEBUG - Response Code: 404 2025-08-14 03:51:31,043 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:31,050 - DEBUG - Response Code: 404 2025-08-14 03:51:32,050 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:32,058 - DEBUG - Response Code: 404 2025-08-14 03:51:33,059 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:33,068 - DEBUG - Response Code: 404 2025-08-14 03:51:34,069 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f40f0938-5b54-4c7c-8ac7-14b360cbabc9 2025-08-14 03:51:34,075 - DEBUG - Response Code: 404 2025-08-14 03:51:34,096 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:51:34,105 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:51:34,106 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:51:34,106 - INFO - VM EVPN_VN1_VM1 is removed in Compute, and routes are removed in all compute nodes 2025-08-14 03:51:34,334 - INFO - Routes for VM EVPN_VN1_VM1 is removed in all control-nodes 2025-08-14 03:51:34,334 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-2:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:51:34,393 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-2?flat 2025-08-14 03:51:34,430 - DEBUG - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:51:34,430 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-14 03:51:34,430 - DEBUG - interface for vm f40f0938-5b54-4c7c-8ac7-14b360cbabc9 deleted from vrouter uve ... 2025-08-14 03:51:34,430 - DEBUG - Deleted interface not in error interface list ... 2025-08-14 03:51:37,431 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-2:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:51:37,509 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-2?flat 2025-08-14 03:51:37,547 - DEBUG - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:51:37,547 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-14 03:51:37,547 - DEBUG - interface for vm f40f0938-5b54-4c7c-8ac7-14b360cbabc9 deleted from vrouter uve ... 2025-08-14 03:51:37,547 - DEBUG - Deleted interface not in error interface list ... 2025-08-14 03:51:40,548 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-2:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:51:40,605 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-2?flat 2025-08-14 03:51:40,641 - DEBUG - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:51:40,641 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-14 03:51:40,641 - DEBUG - interface for vm f40f0938-5b54-4c7c-8ac7-14b360cbabc9 deleted from vrouter uve ... 2025-08-14 03:51:40,641 - DEBUG - Deleted interface not in error interface list ... 2025-08-14 03:51:43,641 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-2:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:51:43,704 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-2?flat 2025-08-14 03:51:43,742 - DEBUG - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:51:43,742 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-14 03:51:43,743 - DEBUG - interface for vm f40f0938-5b54-4c7c-8ac7-14b360cbabc9 deleted from vrouter uve ... 2025-08-14 03:51:43,743 - DEBUG - Deleted interface not in error interface list ... 2025-08-14 03:51:46,744 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3851-2:Compute:contrail-vrouter-agent:0?flat 2025-08-14 03:51:46,807 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3851-2?flat 2025-08-14 03:51:46,843 - DEBUG - VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is not present in vrouter an-jenkins-deploy-platform-ansible-os-3851-2 uve 2025-08-14 03:51:46,843 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-14 03:51:46,843 - DEBUG - interface for vm f40f0938-5b54-4c7c-8ac7-14b360cbabc9 deleted from vrouter uve ... 2025-08-14 03:51:46,843 - DEBUG - Deleted interface not in error interface list ... 2025-08-14 03:51:46,843 - DEBUG - Validated that VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is removed in Vrouter UVE 2025-08-14 03:51:46,843 - DEBUG - Verifying through opserver in 10.0.0.38 2025-08-14 03:51:46,843 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1?flat 2025-08-14 03:51:46,863 - DEBUG - f40f0938-5b54-4c7c-8ac7-14b360cbabc9 BM not in default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 uve 2025-08-14 03:51:46,863 - DEBUG - Validated that VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is not present in the VN UVE 2025-08-14 03:51:46,863 - DEBUG - Verifying the f40f0938-5b54-4c7c-8ac7-14b360cbabc9 virtual network link through opserver 10.0.0.38 2025-08-14 03:51:46,863 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-14 03:51:46,876 - INFO - f40f0938-5b54-4c7c-8ac7-14b360cbabc9 vm uve deleted from opserver 2025-08-14 03:51:46,876 - DEBUG - Verifying the f40f0938-5b54-4c7c-8ac7-14b360cbabc9 virtual network link through opserver 10.0.0.27 2025-08-14 03:51:46,876 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-08-14 03:51:46,890 - INFO - f40f0938-5b54-4c7c-8ac7-14b360cbabc9 vm uve deleted from opserver 2025-08-14 03:51:46,890 - DEBUG - Verifying the f40f0938-5b54-4c7c-8ac7-14b360cbabc9 virtual network link through opserver 10.0.0.23 2025-08-14 03:51:46,890 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-08-14 03:51:46,903 - INFO - f40f0938-5b54-4c7c-8ac7-14b360cbabc9 vm uve deleted from opserver 2025-08-14 03:51:46,904 - INFO - Validated that VM f40f0938-5b54-4c7c-8ac7-14b360cbabc9 is removed from Opserver 2025-08-14 03:51:46,904 - INFO - Deleting VN EVPN-VN1 2025-08-14 03:51:47,084 - DEBUG - Response for deleting network () 2025-08-14 03:51:47,084 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/faec64a8-bd21-4b65-a564-9d0d9a920bcb 2025-08-14 03:51:47,093 - DEBUG - Response Code: 404 2025-08-14 03:51:47,093 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2025-08-14 03:51:47,098 - DEBUG - Requesting: http://10.0.0.38:8082/domain/16362ee5-4ea2-4ad6-8a9c-e34af37c4ada 2025-08-14 03:51:47,158 - DEBUG - Requesting: http://10.0.0.38:8082/project/267caa7c-ab72-42d1-a420-7d6ef32f5c83 2025-08-14 03:51:47,253 - INFO - Validated that VN EVPN-VN1 is not found in API Server 2025-08-14 03:51:47,264 - DEBUG - VN EVPN-VN1 is not present in Agent 10.0.0.65 2025-08-14 03:51:47,276 - DEBUG - VN EVPN-VN1 is not present in Agent 10.0.0.130 2025-08-14 03:51:47,276 - INFO - Validated that VN EVPN-VN1 is not in any agent 2025-08-14 03:51:47,281 - DEBUG - VRF 2 is not seen in agent 10.0.0.65 2025-08-14 03:51:47,286 - DEBUG - Vrouter 10.0.0.65 does not have vrf 2 for VN EVPN-VN1 2025-08-14 03:51:47,292 - DEBUG - VRF 2 is not seen in agent 10.0.0.130 2025-08-14 03:51:47,298 - DEBUG - Vrouter 10.0.0.130 does not have vrf 2 for VN EVPN-VN1 2025-08-14 03:51:47,298 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestEvpnCasesRestart-59825578:EVPN-VN1 2025-08-14 03:51:47,318 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN EVPN-VN1 info 2025-08-14 03:51:49,655 - DEBUG - Skipping xmpp flap check 2025-08-14 03:51:49,655 - INFO - END TEST : test_with_vxlan_encap_agent_restart : PASSED[0:04:35] 2025-08-14 03:51:49,655 - INFO - -------------------------------------------------------------------------------- 2025-08-14 03:51:50,375 - INFO - Deleted project: ctest-TestEvpnCasesRestart-59825578, ID : 267caa7c-ab72-42d1-a420-7d6ef32f5c83