is in ACTIVE state now
2026-01-01 03:14:42,666 - INFO - VM name : ctest-DisablePolicyEcmpIpv6-68577754-24412194
2026-01-01 03:14:42,771 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-68577754-24412194 ID is de1c9068-331d-4a47-bc21-df1ebbd64af8
2026-01-01 03:14:42,771 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-68577754-24412194 launched on Node an-jenkins-deploy-platform-ansible-os-4945-1
2026-01-01 03:14:42,864 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/de1c9068-331d-4a47-bc21-df1ebbd64af8
2026-01-01 03:14:42,874 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/7989a0ee-0fe7-44f4-8081-37c1caebd9cb
2026-01-01 03:14:44,040 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.65 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.481 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.481/2.067/3.653/1.586 ms')
2026-01-01 03:14:44,040 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-DisablePolicyEcmpIpv6-68577754-24412194 passed
2026-01-01 03:14:44,111 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-01-01 03:14:44,112 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-68577754-24412194, IP 237d:f0a7:1de8:3149:67a4:2a8d:0:6, Port 22
2026-01-01 03:14:44,274 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-68577754-24412194 is ready for SSH connections
2026-01-01 03:14:44,342 - INFO - Policy DISABLED on VMI 7989a0ee-0fe7-44f4-8081-37c1caebd9cb
2026-01-01 03:14:44,342 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-01-01 03:14:44,342 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep 237d:f0a7:1de8:3149:67a4:2a8d:0:6 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.242, gateway password: c0ntrail123
2026-01-01 03:14:44,342 - DEBUG - ifconfig -a| grep 237d:f0a7:1de8:3149:67a4:2a8d:0:6 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1
2026-01-01 03:15:21,625 - DEBUG - eth0
2026-01-01 03:15:21,625 - DEBUG - Running remote_cmd, Cmd : sysctl net.ipv6.conf.eth0.accept_dad=0, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.242, gateway password: c0ntrail123
2026-01-01 03:15:21,625 - DEBUG - sysctl net.ipv6.conf.eth0.accept_dad=0
2026-01-01 03:15:22,132 - DEBUG - net.ipv6.conf.eth0.accept_dad = 0
2026-01-01 03:15:22,370 - INFO - Created InterfaceRouteTable ['default-domain', 'ctest-DisablePolicyEcmpIpv6-68577754', 'ctest-intf-rtb-85175296'](UUID 62ed4cc4-ec93-482c-bf18-b81b59ebcdf1), prefixes : ['3384:1cc7:daf5:65e4:8e5d:4078::/96']
2026-01-01 03:15:22,454 - INFO - Added intf route table 62ed4cc4-ec93-482c-bf18-b81b59ebcdf1 to port 7989a0ee-0fe7-44f4-8081-37c1caebd9cb
2026-01-01 03:15:22,454 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.242, gateway password: c0ntrail123
2026-01-01 03:15:22,454 - DEBUG - ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1
2026-01-01 03:15:22,883 - DEBUG - eth0
2026-01-01 03:15:22,883 - DEBUG - Running remote_cmd, Cmd : ifconfig eth0 inet6 add 3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.242, gateway password: c0ntrail123
2026-01-01 03:15:22,884 - DEBUG - ifconfig eth0 inet6 add 3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560
2026-01-01 03:15:23,373 - DEBUG - None
2026-01-01 03:15:23,374 - DEBUG - Running remote_cmd, Cmd : ifconfig -a, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.242, gateway password: c0ntrail123
2026-01-01 03:15:23,374 - DEBUG - ifconfig -a
2026-01-01 03:15:23,863 - DEBUG - eth0 Link encap:Ethernet HWaddr 02:79:89:a0:ee:0f
inet addr:19.242.226.6 Bcast:19.242.226.63 Mask:255.255.255.192
inet6 addr: fe80::79:89ff:fea0:ee0f/64 Scope:Link
inet6 addr: 237d:f0a7:1de8:3149:67a4:2a8d:0:6/64 Scope:Global
inet6 addr: 3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560/128 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:574 errors:0 dropped:0 overruns:0 frame:0
TX packets:509 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:51863 (51.8 KB) TX bytes:43872 (43.8 KB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
2026-01-01 03:15:23,863 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-68577754-80212260
2026-01-01 03:15:24,046 - INFO - Removed intf route table a64657fc-dfd1-4bd3-a81c-01ee9ee9618f from port 5a7be88c-8adb-47a8-92f8-ee29c2bad3fd
2026-01-01 03:15:24,090 - INFO - Deleted Interface route table a64657fc-dfd1-4bd3-a81c-01ee9ee9618f
2026-01-01 03:15:24,129 - INFO - ECMP routes found in agent 10.0.0.23, for prefix 3384:1cc7:daf5:65e4:8e5d:4078::/96
2026-01-01 03:15:24,133 - INFO - Tap interface tap6d664ebc-84 found in ecmp routes in agent 10.0.0.23
2026-01-01 03:15:24,151 - INFO - ECMP routes found in agent 10.0.0.242, for prefix 3384:1cc7:daf5:65e4:8e5d:4078::/96
2026-01-01 03:15:24,153 - INFO - Tap interface tap7989a0ee-0f found in ecmp routes in agent 10.0.0.242
2026-01-01 03:15:24,252 - DEBUG - Executing command: sudo tcpdump -nni tap6d664ebc-84 -U '(udp and src host 237d:f0a7:1de8:3149:67a4:2a8d:0:3 and dst host 3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560 and dst port 11000)' -w /tmp/tap6d664ebc-84_ctest-random-95691052.pcap
2026-01-01 03:15:24,729 - DEBUG - Executing command: sudo tcpdump -nni tap7989a0ee-0f -U '(udp and src host 237d:f0a7:1de8:3149:67a4:2a8d:0:3 and dst host 3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560 and dst port 11000)' -w /tmp/tap7989a0ee-0f_ctest-random-75650755.pcap
2026-01-01 03:15:25,067 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 32130 3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560 11000 < icmp_error, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.242, gateway password: c0ntrail123
2026-01-01 03:15:25,067 - DEBUG - dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 32130 3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560 11000 < icmp_error
2026-01-01 03:15:27,592 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.000912417 s, 110 kB/s
2026-01-01 03:15:27,592 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap6d664ebc-84_ctest-random-95691052.pcap | grep -c length
2026-01-01 03:15:27,608 - DEBUG - STDOUT: 0
2026-01-01 03:15:27,608 - DEBUG - STDERR: reading from file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-01-01 03:15:27,608 - WARNING - No packets are found in tcpdump output file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap but expected some packets
2026-01-01 03:15:29,609 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap6d664ebc-84_ctest-random-95691052.pcap | grep -c length
2026-01-01 03:15:29,622 - DEBUG - STDOUT: 0
2026-01-01 03:15:29,623 - DEBUG - STDERR: reading from file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-01-01 03:15:29,623 - WARNING - No packets are found in tcpdump output file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap but expected some packets
2026-01-01 03:15:31,624 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap6d664ebc-84_ctest-random-95691052.pcap | grep -c length
2026-01-01 03:15:31,641 - DEBUG - STDOUT: 0
2026-01-01 03:15:31,641 - DEBUG - STDERR: reading from file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-01-01 03:15:31,641 - WARNING - No packets are found in tcpdump output file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap but expected some packets
2026-01-01 03:15:33,642 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap6d664ebc-84_ctest-random-95691052.pcap | grep -c length
2026-01-01 03:15:33,657 - DEBUG - STDOUT: 0
2026-01-01 03:15:33,657 - DEBUG - STDERR: reading from file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-01-01 03:15:33,657 - WARNING - No packets are found in tcpdump output file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap but expected some packets
2026-01-01 03:15:35,658 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap6d664ebc-84_ctest-random-95691052.pcap | grep -c length
2026-01-01 03:15:35,672 - DEBUG - STDOUT: 0
2026-01-01 03:15:35,672 - DEBUG - STDERR: reading from file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-01-01 03:15:35,672 - WARNING - No packets are found in tcpdump output file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap but expected some packets
2026-01-01 03:15:37,673 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap6d664ebc-84_ctest-random-95691052.pcap | grep -c length
2026-01-01 03:15:37,688 - DEBUG - STDOUT: 0
2026-01-01 03:15:37,688 - DEBUG - STDERR: reading from file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-01-01 03:15:37,688 - WARNING - No packets are found in tcpdump output file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap but expected some packets
2026-01-01 03:15:39,689 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap6d664ebc-84_ctest-random-95691052.pcap | grep -c length
2026-01-01 03:15:39,704 - DEBUG - STDOUT: 0
2026-01-01 03:15:39,704 - DEBUG - STDERR: reading from file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-01-01 03:15:39,704 - WARNING - No packets are found in tcpdump output file /tmp/tap6d664ebc-84_ctest-random-95691052.pcap but expected some packets
2026-01-01 03:15:39,705 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7989a0ee-0f_ctest-random-75650755.pcap | grep -c length
2026-01-01 03:15:39,718 - DEBUG - STDOUT: 1
2026-01-01 03:15:39,719 - DEBUG - STDERR: reading from file /tmp/tap7989a0ee-0f_ctest-random-75650755.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-01-01 03:15:39,719 - INFO - 1 packets are found in tcpdump output as expected
2026-01-01 03:15:39,719 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2026-01-01 03:15:41,762 - INFO - Tcpdump verification on VM 237d:f0a7:1de8:3149:67a4:2a8d:0:6 passed
2026-01-01 03:15:42,075 - DEBUG - Forward and reverse flows for match {'sip': '237d:f0a7:1de8:3149:67a4:2a8d:0:3', 'dip': '3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560', 'sport': '32130', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0
2026-01-01 03:15:42,075 - DEBUG - Flow verification PASSED on node: 10.0.0.242 for VMs - Sender: 237d:f0a7:1de8:3149:67a4:2a8d:0:3, Receiver: 3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560, Flow count expected: 0 0, got: 0 0
2026-01-01 03:15:42,075 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2026-01-01 03:15:44,084 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2026-01-01 03:15:46,380 - DEBUG - Forward and reverse flows for match {'sip': '237d:f0a7:1de8:3149:67a4:2a8d:0:3', 'dip': '3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560', 'sport': '32130', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0
2026-01-01 03:15:46,381 - DEBUG - Flow verification PASSED on node: 10.0.0.242 for VMs - Sender: 237d:f0a7:1de8:3149:67a4:2a8d:0:3, Receiver: 3384:1cc7:daf5:65e4:8e5d:4078:37ae:b560, Flow count expected: 0 0, got: 0 0
2026-01-01 03:15:46,381 - INFO - Traffic verification for ECMP passed
2026-01-01 03:15:46,381 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-68577754-24412194
2026-01-01 03:15:46,538 - INFO - Removed intf route table 62ed4cc4-ec93-482c-bf18-b81b59ebcdf1 from port 7989a0ee-0fe7-44f4-8081-37c1caebd9cb
2026-01-01 03:15:46,582 - INFO - Deleted Interface route table 62ed4cc4-ec93-482c-bf18-b81b59ebcdf1
2026-01-01 03:15:46,582 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-68577754-54777737
2026-01-01 03:15:46,746 - INFO - Removed intf route table c1cb5619-9498-4631-b0dc-b804e62c131f from port 6d664ebc-84a8-4760-b777-5f2ad08977d3
2026-01-01 03:15:46,782 - INFO - Deleted Interface route table c1cb5619-9498-4631-b0dc-b804e62c131f
2026-01-01 03:15:46,782 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-68577754-13504032
2026-01-01 03:15:46,880 - INFO - Deleting VN ctest-vn-54388491
2026-01-01 03:15:46,934 - DEBUG - VN dbae46ec-6a88-4ae4-b78e-4e7f9becad96 still in use: Unable to complete operation on network dbae46ec-6a88-4ae4-b78e-4e7f9becad96. There are one or more ports still in use on the network.
Neutron server returns request_ids: ['req-b083afa7-06c7-4fbc-ac49-aa61cb25a166']
2026-01-01 03:15:46,934 - WARNING - Deleting VN ctest-vn-54388491 failed..Will retry
2026-01-01 03:15:49,180 - DEBUG - Response for deleting network ()
2026-01-01 03:15:49,180 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/6c09bc25-0be4-48cc-aa3b-7556122651d1
2026-01-01 03:15:49,192 - DEBUG - Response Code: 404
2026-01-01 03:15:49,192 - DEBUG - Requesting: http://10.0.0.38:8082/domains
2026-01-01 03:15:49,198 - DEBUG - Requesting: http://10.0.0.38:8082/domain/b5800509-dc40-4626-b491-badce4b95394
2026-01-01 03:15:49,254 - DEBUG - Requesting: http://10.0.0.38:8082/project/50f35b66-419b-46e7-b29b-932e04edc21d
2026-01-01 03:15:49,351 - INFO - Validated that VN ctest-vn-54388491 is not found in API Server
2026-01-01 03:15:49,366 - DEBUG - VN ctest-vn-54388491 is not present in Agent 10.0.0.242
2026-01-01 03:15:49,373 - DEBUG - VN ctest-vn-54388491 is still found in 10.0.0.23
2026-01-01 03:15:54,384 - DEBUG - VN ctest-vn-54388491 is not present in Agent 10.0.0.242
2026-01-01 03:15:54,394 - DEBUG - VN ctest-vn-54388491 is not present in Agent 10.0.0.23
2026-01-01 03:15:54,394 - INFO - Validated that VN ctest-vn-54388491 is not in any agent
2026-01-01 03:15:54,417 - DEBUG - Routing instance for VN ctest-vn-54388491 is still found in Control-node 10.0.0.249
2026-01-01 03:15:59,442 - DEBUG - Routing instance for VN ctest-vn-54388491 is still found in Control-node 10.0.0.249
2026-01-01 03:16:04,479 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-54388491 info
2026-01-01 03:16:05,790 - ERROR - XMPP status is {'10.0.0.242': {'10.20.0.14': '0', '10.20.0.18': '0'}, '10.0.0.23': {'10.20.0.14': '1', '10.20.0.18': '1'}}, Expected: {'10.0.0.242': {'10.20.0.14': '0', '10.20.0.18': '0'}, '10.0.0.23': {'10.20.0.14': '0', '10.20.0.18': '0'}}
2026-01-01 03:16:05,791 - INFO -
2026-01-01 03:16:05,791 - INFO - END TEST : test_ecmp_with_static_routes : FAILED[0:04:49]
2026-01-01 03:16:05,791 - INFO - --------------------------------------------------------------------------------
2026-01-01 03:16:06,579 - INFO - Deleted project: ctest-DisablePolicyEcmpIpv6-68577754, ID : 50f35b66-419b-46e7-b29b-932e04edc21d