is in ACTIVE state now
2026-02-09 03:22:18,808 - INFO - VM name : ctest-DisablePolicyEcmpIpv6-67946723-11117689
2026-02-09 03:22:18,887 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67946723-11117689 ID is 066b215a-46ad-4c51-b3e8-9288e5661e9a
2026-02-09 03:22:18,888 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67946723-11117689 launched on Node an-jenkins-deploy-platform-ansible-os-5238-1
2026-02-09 03:22:18,961 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/066b215a-46ad-4c51-b3e8-9288e5661e9a
2026-02-09 03:22:18,971 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/4496c384-3b8a-40e1-9b66-398de94babd4
2026-02-09 03:22:20,128 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=2.74 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.471 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.471/1.604/2.738/1.133 ms')
2026-02-09 03:22:20,128 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-DisablePolicyEcmpIpv6-67946723-11117689 passed
2026-02-09 03:22:20,195 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-02-09 03:22:20,196 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-67946723-11117689, IP 2d6f:39d4:4de2:d69:8cee:314a:0:6, Port 22
2026-02-09 03:22:20,368 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67946723-11117689 is ready for SSH connections
2026-02-09 03:22:20,434 - INFO - Policy DISABLED on VMI 4496c384-3b8a-40e1-9b66-398de94babd4
2026-02-09 03:22:20,435 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-02-09 03:22:20,435 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep 2d6f:39d4:4de2:d69:8cee:314a:0:6 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2026-02-09 03:22:20,435 - DEBUG - ifconfig -a| grep 2d6f:39d4:4de2:d69:8cee:314a:0:6 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1
2026-02-09 03:23:00,273 - DEBUG - eth0
2026-02-09 03:23:00,273 - DEBUG - Running remote_cmd, Cmd : sysctl net.ipv6.conf.eth0.accept_dad=0, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2026-02-09 03:23:00,273 - DEBUG - sysctl net.ipv6.conf.eth0.accept_dad=0
2026-02-09 03:23:00,879 - DEBUG - net.ipv6.conf.eth0.accept_dad = 0
2026-02-09 03:23:01,107 - INFO - Created InterfaceRouteTable ['default-domain', 'ctest-DisablePolicyEcmpIpv6-67946723', 'ctest-intf-rtb-14395636'](UUID c1e7ce01-a1b2-45c2-bc78-d97613a86fd5), prefixes : ['2af9:d6ae:7c68:76d:a42b:b233::/96']
2026-02-09 03:23:01,174 - INFO - Added intf route table c1e7ce01-a1b2-45c2-bc78-d97613a86fd5 to port 4496c384-3b8a-40e1-9b66-398de94babd4
2026-02-09 03:23:01,175 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2026-02-09 03:23:01,175 - DEBUG - ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1
2026-02-09 03:23:01,638 - DEBUG - eth0
2026-02-09 03:23:01,638 - DEBUG - Running remote_cmd, Cmd : ifconfig eth0 inet6 add 2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2026-02-09 03:23:01,638 - DEBUG - ifconfig eth0 inet6 add 2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e
2026-02-09 03:23:02,160 - DEBUG - None
2026-02-09 03:23:02,160 - DEBUG - Running remote_cmd, Cmd : ifconfig -a, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2026-02-09 03:23:02,160 - DEBUG - ifconfig -a
2026-02-09 03:23:02,690 - DEBUG - eth0 Link encap:Ethernet HWaddr 02:44:96:c3:84:3b
inet addr:17.234.68.70 Bcast:17.234.68.127 Mask:255.255.255.192
inet6 addr: 2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e/128 Scope:Global
inet6 addr: fe80::44:96ff:fec3:843b/64 Scope:Link
inet6 addr: 2d6f:39d4:4de2:d69:8cee:314a:0:6/64 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:569 errors:0 dropped:0 overruns:0 frame:0
TX packets:499 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:51394 (51.3 KB) TX bytes:43084 (43.0 KB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
2026-02-09 03:23:02,691 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-67946723-73859336
2026-02-09 03:23:02,842 - INFO - Removed intf route table 8b497d20-e204-44e2-a542-5f2cc5fb84e3 from port 1a844a67-e482-48e0-b69c-a75eccb56709
2026-02-09 03:23:02,888 - INFO - Deleted Interface route table 8b497d20-e204-44e2-a542-5f2cc5fb84e3
2026-02-09 03:23:02,901 - INFO - ECMP routes found in agent 10.0.0.65, for prefix 2af9:d6ae:7c68:76d:a42b:b233::/96
2026-02-09 03:23:02,901 - INFO - Tap interface tap8f2907b5-33 found in ecmp routes in agent 10.0.0.65
2026-02-09 03:23:02,913 - INFO - ECMP routes found in agent 10.0.0.27, for prefix 2af9:d6ae:7c68:76d:a42b:b233::/96
2026-02-09 03:23:02,913 - INFO - Tap interface tap4496c384-3b found in ecmp routes in agent 10.0.0.27
2026-02-09 03:23:03,008 - DEBUG - Executing command: sudo tcpdump -nni tap8f2907b5-33 -U '(udp and src host 2d6f:39d4:4de2:d69:8cee:314a:0:3 and dst host 2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e and dst port 11000)' -w /tmp/tap8f2907b5-33_ctest-random-79644858.pcap
2026-02-09 03:23:03,523 - DEBUG - Executing command: sudo tcpdump -nni tap4496c384-3b -U '(udp and src host 2d6f:39d4:4de2:d69:8cee:314a:0:3 and dst host 2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e and dst port 11000)' -w /tmp/tap4496c384-3b_ctest-random-42807487.pcap
2026-02-09 03:23:03,904 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 31722 2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e 11000 < icmp_error, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2026-02-09 03:23:03,904 - DEBUG - dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 31722 2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e 11000 < icmp_error
2026-02-09 03:23:06,451 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.00142229 s, 70.3 kB/s
2026-02-09 03:23:06,451 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap8f2907b5-33_ctest-random-79644858.pcap | grep -c length
2026-02-09 03:23:06,465 - DEBUG - STDOUT: 0
2026-02-09 03:23:06,465 - DEBUG - STDERR: reading from file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-02-09 03:23:06,465 - WARNING - No packets are found in tcpdump output file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap but expected some packets
2026-02-09 03:23:08,466 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap8f2907b5-33_ctest-random-79644858.pcap | grep -c length
2026-02-09 03:23:08,480 - DEBUG - STDOUT: 0
2026-02-09 03:23:08,480 - DEBUG - STDERR: reading from file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-02-09 03:23:08,480 - WARNING - No packets are found in tcpdump output file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap but expected some packets
2026-02-09 03:23:10,481 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap8f2907b5-33_ctest-random-79644858.pcap | grep -c length
2026-02-09 03:23:10,496 - DEBUG - STDOUT: 0
2026-02-09 03:23:10,496 - DEBUG - STDERR: reading from file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-02-09 03:23:10,496 - WARNING - No packets are found in tcpdump output file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap but expected some packets
2026-02-09 03:23:12,497 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap8f2907b5-33_ctest-random-79644858.pcap | grep -c length
2026-02-09 03:23:12,512 - DEBUG - STDOUT: 0
2026-02-09 03:23:12,512 - DEBUG - STDERR: reading from file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-02-09 03:23:12,512 - WARNING - No packets are found in tcpdump output file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap but expected some packets
2026-02-09 03:23:14,512 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap8f2907b5-33_ctest-random-79644858.pcap | grep -c length
2026-02-09 03:23:14,526 - DEBUG - STDOUT: 0
2026-02-09 03:23:14,527 - DEBUG - STDERR: reading from file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-02-09 03:23:14,527 - WARNING - No packets are found in tcpdump output file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap but expected some packets
2026-02-09 03:23:16,527 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap8f2907b5-33_ctest-random-79644858.pcap | grep -c length
2026-02-09 03:23:16,543 - DEBUG - STDOUT: 0
2026-02-09 03:23:16,543 - DEBUG - STDERR: reading from file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-02-09 03:23:16,543 - WARNING - No packets are found in tcpdump output file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap but expected some packets
2026-02-09 03:23:18,544 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap8f2907b5-33_ctest-random-79644858.pcap | grep -c length
2026-02-09 03:23:18,562 - DEBUG - STDOUT: 0
2026-02-09 03:23:18,562 - DEBUG - STDERR: reading from file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-02-09 03:23:18,562 - WARNING - No packets are found in tcpdump output file /tmp/tap8f2907b5-33_ctest-random-79644858.pcap but expected some packets
2026-02-09 03:23:18,562 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap4496c384-3b_ctest-random-42807487.pcap | grep -c length
2026-02-09 03:23:18,576 - DEBUG - STDOUT: 1
2026-02-09 03:23:18,576 - DEBUG - STDERR: reading from file /tmp/tap4496c384-3b_ctest-random-42807487.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-02-09 03:23:18,576 - INFO - 1 packets are found in tcpdump output as expected
2026-02-09 03:23:18,576 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2026-02-09 03:23:20,620 - INFO - Tcpdump verification on VM 2d6f:39d4:4de2:d69:8cee:314a:0:6 passed
2026-02-09 03:23:20,917 - DEBUG - Forward and reverse flows for match {'sip': '2d6f:39d4:4de2:d69:8cee:314a:0:3', 'dip': '2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e', 'sport': '31722', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0
2026-02-09 03:23:20,918 - DEBUG - Flow verification PASSED on node: 10.0.0.27 for VMs - Sender: 2d6f:39d4:4de2:d69:8cee:314a:0:3, Receiver: 2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e, Flow count expected: 0 0, got: 0 0
2026-02-09 03:23:20,918 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2026-02-09 03:23:22,926 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2026-02-09 03:23:25,229 - DEBUG - Forward and reverse flows for match {'sip': '2d6f:39d4:4de2:d69:8cee:314a:0:3', 'dip': '2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e', 'sport': '31722', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0
2026-02-09 03:23:25,229 - DEBUG - Flow verification PASSED on node: 10.0.0.27 for VMs - Sender: 2d6f:39d4:4de2:d69:8cee:314a:0:3, Receiver: 2af9:d6ae:7c68:76d:a42b:b233:b2c2:df3e, Flow count expected: 0 0, got: 0 0
2026-02-09 03:23:25,229 - INFO - Traffic verification for ECMP passed
2026-02-09 03:23:25,229 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-67946723-11117689
2026-02-09 03:23:25,386 - INFO - Removed intf route table c1e7ce01-a1b2-45c2-bc78-d97613a86fd5 from port 4496c384-3b8a-40e1-9b66-398de94babd4
2026-02-09 03:23:25,423 - INFO - Deleted Interface route table c1e7ce01-a1b2-45c2-bc78-d97613a86fd5
2026-02-09 03:23:25,423 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-67946723-81895630
2026-02-09 03:23:25,578 - INFO - Removed intf route table 11a91bd6-194e-4426-83c0-6f13c006774f from port 8f2907b5-3316-4e9a-b75c-d37ca1570a8f
2026-02-09 03:23:25,612 - INFO - Deleted Interface route table 11a91bd6-194e-4426-83c0-6f13c006774f
2026-02-09 03:23:25,612 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-67946723-92662148
2026-02-09 03:23:25,688 - INFO - Deleting VN ctest-vn-69376742
2026-02-09 03:23:25,735 - DEBUG - VN 7ccdb5f8-1d9b-4320-8a0c-dc5cbd845b24 still in use: Unable to complete operation on network 7ccdb5f8-1d9b-4320-8a0c-dc5cbd845b24. There are one or more ports still in use on the network.
Neutron server returns request_ids: ['req-6f91a3f9-8b3a-48f0-948b-971a1ceafec1']
2026-02-09 03:23:25,735 - WARNING - Deleting VN ctest-vn-69376742 failed..Will retry
2026-02-09 03:23:27,909 - DEBUG - Response for deleting network ()
2026-02-09 03:23:27,909 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/96f1ff9d-4eed-402a-8d5e-bcae52cea627
2026-02-09 03:23:27,915 - DEBUG - Response Code: 404
2026-02-09 03:23:27,915 - DEBUG - Requesting: http://10.0.0.19:8082/domains
2026-02-09 03:23:27,920 - DEBUG - Requesting: http://10.0.0.19:8082/domain/069df225-a7cb-4796-a8d5-681a22d1bf82
2026-02-09 03:23:27,933 - DEBUG - Requesting: http://10.0.0.19:8082/project/8836169f-7c5f-4768-95e5-4ba3f73d0cc5
2026-02-09 03:23:28,017 - INFO - Validated that VN ctest-vn-69376742 is not found in API Server
2026-02-09 03:23:28,029 - DEBUG - VN ctest-vn-69376742 is not present in Agent 10.0.0.27
2026-02-09 03:23:28,040 - DEBUG - VN ctest-vn-69376742 is not present in Agent 10.0.0.65
2026-02-09 03:23:28,040 - INFO - Validated that VN ctest-vn-69376742 is not in any agent
2026-02-09 03:23:28,068 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-69376742 info
2026-02-09 03:23:29,340 - DEBUG - No XMPP flaps were noticed during the test
2026-02-09 03:23:29,340 - INFO - END TEST : test_ecmp_with_static_routes : PASSED[0:04:34]
2026-02-09 03:23:29,340 - INFO - --------------------------------------------------------------------------------
2026-02-09 03:23:30,039 - INFO - Deleted project: ctest-DisablePolicyEcmpIpv6-67946723, ID : 8836169f-7c5f-4768-95e5-4ba3f73d0cc5