Fortinet
This tutorial provides information and examples of how to configure Cloudflare Magic WAN with IPsec tunnels in conjunction with Fortinet FortiGate firewalls.
The FortiGate configuration settings presented here support bidirectional health checks as required by Cloudflare Magic WAN. However, they do not factor in any other traffic flows outside of the tunnel health checks. The configuration may need to be adjusted based on your current FortiGate configuration.
Testing Environment
The FortiGate configuration was tested on two different FortiGate firewalls:
- FortiGate Virtual Appliance version 7.0.8, running on VMware ESXi 6.5
- FortiGate FG80F, version 7.0.12
Magic WAN configuration
The first step to setting up Magic WAN is to add Magic WAN IPsec tunnels and Magic static routes to your Cloudflare account via the dashboard or API.
Before proceeding, ensure that you have the Anycast IPs associated with your account. Check with your Cloudflare account team if you do not yet have them.
Magic IPsec Tunnels
Cloudflare recommends customers configure two Magic IPsec tunnels per firewall/router — one to each of the two Anycast IP addresses.
Go to the Cloudflare dashboard and select your account.
Go to Magic WAN > Configuration.
From the Tunnels tab, select Create.
For the first IPsec tunnel, ensure the following settings are defined (refer to Add tunnels to learn about settings not mentioned here):
- Customer Endpoint: Enter your external/egress interface of the firewall.
- Cloudflare Endpoint: Enter the first of your two Anycast IPs.
- Health check rate: Low.
- Health check type: Reply.
- Health check target: Custom.
- Target address: The target address for the first tunnel is always
172.64.240.253
. - Pre-shared key: Enter your own key or allow Cloudflare to define the key. Refer to Add IPsec tunnel for more information.
For the second tunnel, make the same changes as you did for the first tunnel (including creating a pre-shared key), and ensure the following additional settings are defined:
- Cloudflare Endpoint: Enter the second of your two Anycast IPs.
- Health check target: Custom.
- Target address:
172.64.240.254
.
Magic static routes
Add two Magic static routes to define the IP address space that exists behind the Magic IPsec tunnels — one to each of the two Magic IPsec tunnels defined in the previous section.
By default, the Magic static routes are defined with the priority set to 100
. Cloudflare leverages Equal Cost Multipath Routing (ECMP) and will load balance the traffic equally across the two tunnels. If you prefer to use an Active/Passive model, you can leave the default value for the first route set to 100
, and set the value for the second tunnel to 150
(higher value is a lower priority).
Go to the Cloudflare dashboard and select your account.
Go to Magic WAN > Configuration.
From the Static Routes tab, select Create.
For the first route, ensure the following settings are defined (refer to Configure static routes to learn about settings not mentioned here):
- Prefix: Specify the RFC1918 subnet that exists behind the first Magic IPsec tunnel you have defined in the previous section.
- Tunnel/Next hop: Select your first tunnel (Tunnel 01 of 02).
- Priority: Leave the default value (
100
). - Weight: Leave empty.
- Region code: Leave this set to
All Regions
.
For the second route, ensure the following settings are defined:
- Prefix: Specify the RFC1918 subnet that exists behind the second Magic IPsec tunnel defined in the previous section.
- Tunnel/Next hop: Select your second tunnel (Tunnel 02 of 02).
- Priority: Leave the default value (
100
). - Weight: Leave empty.
- Region code: Leave this set to
All Regions
.
Fortinet FortiGate configuration
Enable asymmetric routing
To ensure health checks work as expected, enable asymmetric routing for ICMP. This option is required. Otherwise, the tunnel health checks which are critical for proper Magic WAN functionality will not work as designed.
Note that enabling asymmetric routing will affect FortiGate behavior. To learn more, refer to How FortiGate behaves when asymmetric routing is enabled.
config system settings set asymroute-icmp enableend
Disable anti-replay protection
For route-based IPsec configurations, you will need to disable anti-replay protection. The command below disables anti-replay protection globally, but you can also do this per firewall policy. Refer to Fortinet’s documentation on anti-replay support per policy to learn more.
config system global set anti-replay disableend
IPsec tunnels
Magic IPsec tunnels leverage a route-based site-to-site VPN model. This model relies on the use of virtual tunnel interfaces and routing to define the traffic that flows across the IPsec tunnels.
Configure two IPsec tunnels using the phase1-interface
and phase2-interface
objects.
The following examples assume wan1
is the external/egress interface of the FortiGate firewall.
Add Phase 1 interfaces
MWAN_IPsec_Tun1
corresponds to Tunnel 01 of 02 added earlier in the Cloudflare section of the configuration. MWAN_IPsec_Tun2
corresponds to Tunnel 02 of 02 added earlier in the Cloudflare section of the configuration.
fortigate # config vpn ipsec phase1-interface edit "MWAN_IPsec_Tun1" set interface "wan1" set ike-version 2 set keylife 28800 set peertype any set net-device enable set proposal aes256gcm-prfsha512 aes256gcm-prfsha384 aes256gcm-prfsha256 set localid "f1473dXXXXXXX72e33.49561179.ipsec.cloudflare.com" set dhgrp 14 set nattraversal disable set remote-gw 162.159.67.210 set add-gw-route enable set psksecret <YOUR_PRE-SHARED_KEY> next edit "MWAN_IPsec_Tun2" set interface "wan1" set ike-version 2 set keylife 28800 set peertype any set net-device enable set proposal aes256gcm-prfsha512 aes256gcm-prfsha384 aes256gcm-prfsha256 set localid "de91565XXXXXXXfbbd6632.49561179.ipsec.cloudflare.com" set dhgrp 14 set nattraversal disable set remote-gw 172.XX.XX.210 set add-gw-route enable set psksecret ENC <YOUR_PRE-SHARED_KEY> nextend
Add Phase 2 interfaces
Add two phase2-interfaces
— one for each of the two phase1-interfaces
as follows:
fortigate # config vpn ipsec phase2-interface edit "MWAN_IPsec_Tun1" set phase1name "MWAN_IPsec_Tun1" set proposal aes256gcm aes128gcm set dhgrp 14 set replay disable set keylifeseconds 3600 set auto-negotiate enable set keepalive enable next edit "MWAN_IPsec_Tun2" set phase1name "MWAN_IPsec_Tun2" set proposal aes256gcm aes128gcm set dhgrp 14 set replay disable set keylifeseconds 3600 set auto-negotiate enable set keepalive enable nextend
Network interfaces
Virtual tunnel interfaces
Configure the virtual tunnel interfaces that were automatically added when specifying the set net-device enable
within the phase1-interface
settings.
These are the only settings that should need to be added to the virtual tunnel interfaces:
ip
: The local IP address (specify with a/32
netmask —255.255.255.255
).remote-ip
: The value associated with the interface address specified earlier in the Magic IPsec tunnels section (specify with a/31
netmask —255.255.255.254
).alias
: This value is optional.
The following examples assume wan1
is the external/egress interface of the FortiGate firewall.
fortigate # config system interface edit "MWAN_IPsec_Tun1" set vdom "root" set ip 10.252.2.91 255.255.255.255 set allowaccess ping set type tunnel set remote-ip 10.252.2.90 255.255.255.254 set alias "MWAN_IPsec_Tun1" set snmp-index 17 set interface "wan1" next edit "MWAN_IPsec_Tun2" set vdom "root" set ip 10.252.2.93 255.255.255.255 set allowaccess ping set type tunnel set remote-ip 10.252.2.92 255.255.255.254 set alias "MWAN_IPsec_Tun2" set snmp-index 18 set interface "wan1" nextend
Validate communication across virtual tunnel interfaces
Once the virtual tunnel interfaces have been configured, you should be able to ping the IP address associated with the remote-ip
attribute.
Below you have examples of a successful result from pinging across both of the virtual tunnel interfaces configured in the previous section:
MWAN_IPsec_Tun1
fortigate # execute ping 10.252.2.90PING 10.252.2.90 (10.252.2.90): 56 data bytes64 bytes from 10.252.2.90: icmp_seq=0 ttl=64 time=5.8 ms64 bytes from 10.252.2.90: icmp_seq=1 ttl=64 time=5.8 ms64 bytes from 10.252.2.90: icmp_seq=2 ttl=64 time=5.8 ms64 bytes from 10.252.2.90: icmp_seq=3 ttl=64 time=5.8 ms64 bytes from 10.252.2.90: icmp_seq=4 ttl=64 time=5.7 ms
--- 10.252.2.90 ping statistics ---5 packets transmitted, 5 packets received, 0% packet lossround-trip min/avg/max = 5.7/5.7/5.8 ms
MWAN_IPsec_Tun2
fortigate # execute ping 10.252.2.92PING 10.252.2.92 (10.252.2.92): 56 data bytes64 bytes from 10.252.2.92: icmp_seq=0 ttl=64 time=6.1 ms64 bytes from 10.252.2.92: icmp_seq=1 ttl=64 time=6.1 ms64 bytes from 10.252.2.92: icmp_seq=2 ttl=64 time=6.1 ms64 bytes from 10.252.2.92: icmp_seq=3 ttl=64 time=6.1 ms64 bytes from 10.252.2.92: icmp_seq=4 ttl=64 time=6.0 ms
--- 10.252.2.92 ping statistics ---5 packets transmitted, 5 packets received, 0% packet lossround-trip min/avg/max = 6.0/6.0/6.1 ms
Zone objects (optional)
This sample configuration assumes there are three zones configured on the FortiGate firewall. These zone objects are used in the policies referenced later in this document:
Trust_Zone
: Contains the LAN interface(s).Untrust_Zone
: Contains the WAN interface.Cloudflare_Zone
: Contains both IPsec Tunnel interfaces.
fortigate # config system zone edit "Cloudflare_Zone" set intrazone allow set interface "MWAN_IPsec_Tun1" "MWAN_IPsec_Tun2" next edit "Trust_Zone" set intrazone allow set interface "internal" next edit "Untrust_Zone" set intrazone allow set interface "wan1" nextend
Create Address Objects
Create Address Objects to represent the Cloudflare IPv4 address space as well as objects for the bidirectional health check Anycast IPs:
config firewall address edit "Cloudflare_IPv4_01" set color 9 set subnet 173.245.48.0 255.255.240.0 next edit "Cloudflare_IPv4_02" set color 9 set subnet 103.21.244.0 255.255.252.0 next edit "Cloudflare_IPv4_03" set color 9 set subnet 103.22.200.0 255.255.252.0 next edit "Cloudflare_IPv4_04" set color 9 set subnet 103.31.4.0 255.255.252.0 next edit "Cloudflare_IPv4_05" set color 9 set subnet 141.101.64.0 255.255.192.0 next edit "Cloudflare_IPv4_06" set color 9 set subnet 108.162.192.0 255.255.192.0 next edit "Cloudflare_IPv4_07" set color 9 set subnet 190.93.240.0 255.255.240.0 next edit "Cloudflare_IPv4_08" set color 9 set subnet 188.114.96.0 255.255.240.0 next edit "Cloudflare_IPv4_09" set color 9 set subnet 197.234.240.0 255.255.252.0 next edit "Cloudflare_IPv4_10" set color 9 set subnet 198.41.128.0 255.255.128.0 next edit "Cloudflare_IPv4_11" set color 9 set subnet 162.158.0.0 255.254.0.0 next edit "Cloudflare_IPv4_12" set color 9 set subnet 104.16.0.0 255.248.0.0 next edit "Cloudflare_IPv4_13" set color 9 set subnet 104.24.0.0 255.252.0.0 next edit "Cloudflare_IPv4_14" set color 9 set subnet 172.64.0.0 255.248.0.0 next edit "Cloudflare_IPv4_15" set color 9 set subnet 131.0.72.0 255.255.252.0 next edit "Bidirect_HC_Endpoint_01" set comment "Bidirectional health check endpoint address" set color 9 set subnet 172.64.240.253 255.255.255.255 next edit "Bidirect_HC_Endpoint_02" set comment "Bidirectional health check endpoint address" set color 9 set subnet 172.64.240.254 255.255.255.255 nextend
Configure Address Group Object
Create an Address Object that contains all of the Cloudflare IPv4 subnets as specified in the previous section. You can copy/paste the CLI commands below into an SSH terminal and the objects will get created automatically:
config firewall addrgrp edit "Cloudflare_IPv4_Nets" set member "Cloudflare_IPv4_01" "Cloudflare_IPv4_02" "Cloudflare_IPv4_03" "Cloudflare_IPv4_04" "Cloudflare_IPv4_05" "Cloudflare_IPv4_06" "Cloudflare_IPv4_07" "Cloudflare_IPv4_08" "Cloudflare_IPv4_09" "Cloudflare_IPv4_10" "Cloudflare_IPv4_11" "Cloudflare_IPv4_12" "Cloudflare_IPv4_13" "Cloudflare_IPv4_14" "Cloudflare_IPv4_15" set color 9 nextend
Add security policy
Add a firewall rule to permit the ICMP traffic associated with the reply style bidirectional health checks.
fortigate (policy) # showconfig firewall policy edit 2 set name "CF_Magic_Health_Checks" set uuid 80eb76ce-3033-51ee-c5e5-d5a670dff3b3 set srcintf "Cloudflare_Zone" set action accept set srcaddr "Cloudflare_IPv4_Nets" set dstaddr "Bidirect_HC_Endpoint_01" "Bidirect_HC_Endpoint_02" set schedule "always" set service "ALL_ICMP" set logtraffic all nextend
Policy-based routing
Policy-based routing rules are required to ensure the traffic associated with the bidirectional health checks received over a Magic IPsec tunnel is sent back across the same Magic IPsec tunnel.
Add two policy-based routing rules, one for each of the two Magic IPsec tunnels.
fortigate # config router policy edit 1 set input-device "MWAN_IPsec_Tun1" set srcaddr "all" set dstaddr "all" set gateway 10.252.2.90 set output-device "MWAN_IPsec_Tun1" next edit 2 set input-device "MWAN_IPsec_Tun2" set srcaddr "all" set dstaddr "all" set gateway 10.252.2.92 set output-device "MWAN_IPsec_Tun2" nextend
Monitor Cloudflare Magic IPsec tunnel health checks
The Cloudflare dashboard monitors the health of all Anycast tunnels on your account that route traffic from Cloudflare to your origin network.
The dashboard shows the view of tunnel health as measured from each Cloudflare location where your traffic is likely to land. If the tunnels are healthy on your side, you will see the majority of servers reporting an up status. It is normal for a subset of these locations to show tunnel status as degraded or unhealthy, since the Internet is not homogeneous and intermediary path issues between Cloudflare and your network can cause interruptions for specific paths.
Not all data centers will be relevant to you at all times. You can refer to the Average ingress traffic (last hour) column to understand if a given data center is receiving traffic for your network, and if its health status is relevant to you.
To check for Anycast tunnel health:
- Go to the Cloudflare dashboard and select your account.
- Go to Magic WAN > Tunnel health.
- In Cloudflare colos, you can choose one or more Cloudflare data centers to filter out the traffic that shows up in your Anycast tunnels. For example, if you chose the Lisbon data center, your Anycast tunnels would only show connections to that data center.
- Below, you have a list of all your Anycast tunnels, as well as their current health status. Find the tunnel you wish to inspect and select the arrow (>) before it to open its details.
- The details pane shows the connection status between different Cloudflare servers and your tunnel. Select Traceroute for details in one of the Cloudflare servers shown to check for issues between Cloudflare and your origin network.
Troubleshooting
Packet Capture
Packet captures allow you to determine whether or not the policy-based routing rules are working as expected.
The expected behavior should look like the examples below — traffic ingressing Tunnel 01 of 02 should egress the same tunnel.
fortigate # diagnose sniffer packet any 'host 172.64.240.253' 4interfaces=[any]filters=[host 172.64.240.253]0.601569 MWAN_IPsec_Tun1 in 172.64.240.253 -> 162.158.176.118: icmp: echo reply0.601585 MWAN_IPsec_Tun1 out 172.64.240.253 -> 162.158.176.118: icmp: echo reply0.611164 MWAN_IPsec_Tun1 in 172.64.240.253 -> 172.71.87.94: icmp: echo reply0.611178 MWAN_IPsec_Tun1 out 172.64.240.253 -> 172.71.87.94: icmp: echo reply0.617562 MWAN_IPsec_Tun1 in 172.64.240.253 -> 172.71.129.214: icmp: echo reply0.617574 MWAN_IPsec_Tun1 out 172.64.240.253 -> 172.71.129.214: icmp: echo reply0.622042 MWAN_IPsec_Tun1 in 172.64.240.253 -> 172.69.61.43: icmp: echo reply0.622056 MWAN_IPsec_Tun1 out 172.64.240.253 -> 172.69.61.43: icmp: echo reply0.624092 MWAN_IPsec_Tun1 in 172.64.240.253 -> 172.68.9.214: icmp: echo reply
Conversely, traffic ingressing Tunnel 02 of 02 should egress the same tunnel:
fortigate # diagnose sniffer packet any 'host 172.64.240.254' 4interfaces=[any]filters=[host 172.64.240.254]0.912041 MWAN_IPsec_Tun2 in 172.64.240.254 -> 172.70.177.56: icmp: echo reply0.912057 MWAN_IPsec_Tun2 out 172.64.240.254 -> 172.70.177.56: icmp: echo reply0.913579 MWAN_IPsec_Tun2 in 172.64.240.254 -> 172.70.221.154: icmp: echo reply0.913592 MWAN_IPsec_Tun2 out 172.64.240.254 -> 172.70.221.154: icmp: echo reply0.914247 MWAN_IPsec_Tun2 in 172.64.240.254 -> 162.158.1.85: icmp: echo reply0.914260 MWAN_IPsec_Tun2 out 172.64.240.254 -> 162.158.1.85: icmp: echo reply0.918533 MWAN_IPsec_Tun2 in 172.64.240.254 -> 172.71.125.75: icmp: echo reply0.918550 MWAN_IPsec_Tun2 out 172.64.240.254 -> 172.71.125.75: icmp: echo reply0.924465 MWAN_IPsec_Tun2 in 172.64.240.254 -> 172.69.21.134: icmp: echo reply
Flow Debugging
Flow debugging can be helpful when it comes to determining whether or not traffic is ingressing/egressing the firewall via the expected path. It takes steps much further than the sniffer packet captures in the previous section, but it creates a tremendous amount of logging and should only be enabled when absolutely necessary.
Additionally, customers will likely need to contact Fortinet technical support for assistance with interpreting the flow debug logs, as well as to obtain recommendations in terms of how to configure FortiGate to ensure flows are routed correctly based on the application’s requirements.
fortigate # diagnose debug disablefortigate # diagnose debug flow filter clearfortigate # diagnose debug resetfortigate # diagnose debug flow filter addr 172.64.240.253fortigate # diagnose debug show flow show function-name enablefortigate # diagnose debug config-error-log timestamps enablefortigate # diagnose debug flow trace start 999fortigate # diagnose debug enable
fortigate # 2023-08-01 09:27:26 id=20085 trace_id=2871 func=print_pkt_detail line=5844 msg="vd-root:0 received a packet(proto=1, 172.64.240.253:56968->172.70.121.28:0) tun_id=162.159.67.210 from MWAN_IPsec_Tun1. type=0, code=0, id=56968, seq=0."2023-08-01 09:27:26 id=20085 trace_id=2871 func=rpdb_srv_match_input line=1036 msg="Match policy routing id=1: to 10.252.2.90 via ifindex-34"2023-08-01 09:27:26 id=20085 trace_id=2871 func=vf_ip_route_input_common line=2605 msg="find a route: flag=00000000 gw-162.159.67.210 via MWAN_IPsec_Tun1"2023-08-01 09:27:26 id=20085 trace_id=2871 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface MWAN_IPsec_Tun1, tun_id=0.0.0.0"2023-08-01 09:27:26 id=20085 trace_id=2871 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel MWAN_IPsec_Tun1"2023-08-01 09:27:26 id=20085 trace_id=2871 func=esp_output4 line=844 msg="IPsec encrypt/auth"2023-08-01 09:27:26 id=20085 trace_id=2871 func=ipsec_output_finish line=544 msg="send to 172.71.91.34 via intf-wan1"2023-08-01 09:27:26 id=20085 trace_id=2872 func=print_pkt_detail line=5844 msg="vd-root:0 received a packet(proto=1, 172.64.240.253:18685->162.158.209.64:0) tun_id=162.159.67.210 from MWAN_IPsec_Tun1. type=0, code=0, id=18685, seq=0."2023-08-01 09:27:26 id=20085 trace_id=2872 func=rpdb_srv_match_input line=1036 msg="Match policy routing id=1: to 10.252.2.90 via ifindex-34"2023-08-01 09:27:26 id=20085 trace_id=2872 func=vf_ip_route_input_common line=2605 msg="find a route: flag=00000000 gw-162.159.67.210 via MWAN_IPsec_Tun1"2023-08-01 09:27:26 id=20085 trace_id=2872 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface MWAN_IPsec_Tun1, tun_id=0.0.0.0"2023-08-01 09:27:26 id=20085 trace_id=2872 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel MWAN_IPsec_Tun1"2023-08-01 09:27:26 id=20085 trace_id=2872 func=esp_output4 line=844 msg="IPsec encrypt/auth"2023-08-01 09:27:26 id=20085 trace_id=2872 func=ipsec_output_finish line=544 msg="send to 172.71.91.34 via intf-wan1"
Disable Flow Debugging
The typical use of CTRL C will not stop Flow Debugging.
You can disable Flow Debugging simply by typing the following at any point while the debug logs are scrolling by:
fortigate # diagnose debug disable