Client-Side SD-WAN with IPsec VPN Deployment Scenario – Expert

Facebooktwittergoogle_plusredditpinterestlinkedinFacebooktwittergoogle_plusredditpinterestlinkedin

This advanced deployment scenario provides a high-level picture of how to combine SD-WAN, IPsec VPN, and BGP routing to provide a branch office with redundant connections to two remote data centers and the networks behind them. Using this deployment scenario allows you to replace private or MPLS connections to data centers with lower-cost encrypted SD-WAN connections over the Internet.

This scenario is intended for network engineers who are familiar with the FortiGate platform and are looking for an example FortiOS 5.6 SD-WAN configuration. It does not include all of the required configuration steps but the intention is to provide the information you need to implement SD-WAN technology.

Configuring the data center FortiGates

The configuration described here must be set up on Data Center 1 FortiGate and Data Center 2 FortiGate. The following steps show how to configure Data Center 1 FortiGate (as shown in the diagram). You can repeat this configuration for Data Center 2 FortiGate, substituting the proper IP addresses and interface names.

This configuration has the following objectives:

  • Zero touch IPsec VPN provisioning of new branches
  • Point-to-multipoint IPsec VPN
  • Central management of data center access from each data center firewall
  • Dynamic peering to share routing information between each branch and the data center

Each data center configuration includes dynamic (or dial-up) IPsec VPN, BGP, firewall policies to control access, and a blackhole route for each branch office.

1. Creating the data center side of the IPsec VPN

To facilitate zero touch provisioning of new spokes to establish VPNs on each data center FortiGate, this example uses dial-up VPNs with auto-discovery-sender enabled in the ADVPN configuration.

Also, add-route is disabled to support multiple dynamic tunnels to the same host advertising the same network. This dynamic discovery of the network is facilitated by the BGP configuration.

Wildcard security associations are used for phase 2 since BGP routes determine whether traffic is sent over the IPSec VPN tunnel. In this example, IPsec VPN is added to each FortiGate interface connected to the Internet.

 

The Phase 1 configuration includes:

  • A dynamic VPN tunnel name that is 11 characters or less.
  • Setting type to dynamic
  • Setting interface to the Internet connected interface
  • Setting peertype to any
  • Setting add-route to disable
  • Setting auto-discovery-sender to enable
 
config vpn ipsec phase1-interface
    edit "vpn-br1-1"
        set type dynamic
        set interface "vlan-3510"
        set peertype any
        set proposal aes256-sha256
        set add-route disable
        set dhgrp 5
        set auto-discovery-sender enable
        set psksecret <password>
    next
    edit "vpn-br1-2"
        set type dynamic
        set interface "vlan-3511"
        set peertype any
        set proposal aes256-sha256
        set add-route disable
        set dhgrp 5
        set auto-discovery-sender enable
        set psksecret <password>
    end
 

The Phase 2 configuration includes:

  • Setting phase1name to the name of the phase 1 configuration
  • Disabling pfs and replay
 
config vpn ipsec phase2-interface
    edit "vpn-br1-1_p2"
        set phase1name "vpn-isp-a"
        set proposal aes256-sha256
        set pfs disable
        set replay disable
     next
     edit "vpn-br1-2_p2"
        set phase1name "vpn-isp-b"
        set proposal aes256-sha256
        set pfs disable
        set replay disable
     end
 
 

2. Adding addresses to the tunnel interfaces

The BGP configuration requires IP addresses assigned to the IPsec VPN tunnel interfaces that BGP peers over. The ADVPN feature enabled by set auto-discovery-sender enable allows FortiOS to establish a point-to-multipoint connection to each FortiGate.

The IPsec VPN tunnel interface ip is set to the IP address that the tunnels will connect to, and remote-ip is set to the highest unused IP address that is part of your tunnel network. This adds two host-based routes to the FortiGate’s routing table that point directly back to the branch FortiGate.

 

The IPsec VPN interface configuration includes:

  • Setting the ip to <vpn interface ip> 255.255.255.255
  • Setting type to tunnel
  • Setting remote-ip to the highest unused IP address in the VPN subnet
  • Setting allowaccess to ping to allow for confirmation that a point-to-point tunnel has been established between the data center FortiGate and the branch FortiGate.
 
config system interface
    edit "vpn-br1-1"
        set vdom "root"
        set ip 10.254.0.1 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.254.0.254
        set interface "port1"
    next
    edit "vpn-br1-2"
        set vdom "root"
        set ip 10.254.1.1 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.254.1.254
        set interface "port2"
    end

3. Implementing route discovery with BGP

Network route discovery is facilitated by BGP and EBGP, which prevent the redistribution of routes learned that are contained in the same autonomous system number as the host. Also, EBGP influences route selection on the branches because of AS-Path prepending.

Enable ebgp-multipath to allow the FortiGate to dynamically discover multiple paths for networks advertised at branches.

Configure neighbor-range and neighbor-group to allow peering relationships to be established without defining each individual peer. The branch IPsec VPN tunnel interface addresses must be in the BGP peer range.

The BGP configuration includes:

  • Enabling ebgp-multipath
  • Enabling soft-reconfiguration and link-down-failover for each BGP peer in the neighbor group
  • Adding the branch remote-as (which is 65501) to each peer configuration
  • Setting the prefix for the neighbor range to the network matching the BGP peers
  • Configuring a network with the prefix of the network advertised into BGP

To facilitate the fastest route failovers, the following timers are set to their lowest values:

  • scan-time
  • advertisement-interval
  • keep-alive timer
  • holdtime-timer
 
config router bgp
    set as 65500
    set router-id 10.10.0.1
    set ebgp-multipath enable
    set scan-time 5
    set graceful-restart enable
    config neighbor-group
        edit "branch-peers-1"
            set advertisement-interval 1
            set link-down-failover enable
            set soft-reconfiguration enable
            set remote-as 65501
            set keep-alive-timer 1
            set holdtime-timer 3
        next
        edit "branch-peers-2"
            set advertisement-interval 1
            set link-down-failover enable
            set soft-reconfiguration enable
            set remote-as 65501
            set keep-alive-timer 1
            set holdtime-timer 3
        next
    end
    config neighbor-range
        edit 1
            set prefix 10.254.0.0 255.255.255.0
            set neighbor-group "branch-peers-1"
        next
        edit 2
            set prefix 10.254.1.0 255.255.255.0
            set neighbor-group "branch-peers-2"
        next
    end
    config network
        edit 1
            set prefix 10.200.1.0 255.255.255.0
        next
        edit 2
            set prefix 10.200.0.0 255.255.255.0
        next
        edit 3
            set prefix 10.200.3.0 255.255.255.0
        next
    end
end

4. Controlling access to data center networks

Create firewall policies to allow users on the branch office networks to access the data center networks (behind the FortiGate). Security profiles can be added to these firewall policies to inspect of layer 7 traffic.

Include a policy on the data center FortiGate to allow a branch FortiGate to check the health of the data center FortiGate by allowing the branch FortiGate to ping the data center FortiGate IPsec VPN interface:

  • Source interface: IPsec VPN interface
  • Destination interface: Internal interface
  • Source Address: Tunnel IP addresses of branch
  • Destination Address: Data Center 1 FortiGate Internal interface
  • Action: Accept
  • Schedule: Always
  • Service: ICMP

Policies to allow traffic from branch networks to reach data center networks should have the following firewall settings:

  • Source interface: IPsec VPN interface
  • Destination interface: Internal interface
  • Source Address: Branch networks
  • Destination Address: Date center networks
  • Action: Accept
  • Schedule: Always (or define a more restrictive schedule)
  • Service: Allowed Service(s)

5. Pointing to branch offices with black hole routes

It is a best practice to create black hole routes with destinations set to each branch  network. If the FortiGate temporarily loses connectivity with a branch network, traffic destined to that network is sent to the black hole until connectivity has been restored.

Each Black hole route includes:

  • Setting dst to the branch network IP address
  • Setting the distance to 255
 
config router static
    edit 1
        set dst 10.0.0.0/14
        set distance 255
        set blackhole enable
    next
end

Configuring Branch FortiGate

The following steps describe how to use the SD-WAN feature to set up the branch FortiGate with redundant connections to the two data centers. This configuration includes the following:

  • Client-side SD-WAN (intelligent load balancing based on link quality)
  • A configuration template for quick deployment of branch FortiGates
  • Split tunneling for Internet access from the branch office networks

The branch FortiGate configuration includes IPsec VPN, BGP, SD-WAN load balancing, and firewall policies to control access.

1. Creating the branch side of the IPsec VPN

The IPsec VPN configuration is similar to a normal site-to-site VPN configuration. Wildcard security associations are used for phase 2 since BGP routes determine whether traffic is sent over the IPSec VPN tunnel.

Create two Phase 1 configurations, one for each data center. These configurations include:

  • Setting peertype to any
  • Setting remote-gw to the IP address of the data center.
 
config vpn ipsec phase1-interface
    edit "vpn_dc1-1"
        set interface "vlan-3000"
        set peertype any
        set proposal aes256-sha256
        set dhgrp 5
        set remote-gw 172.20.10.10
        set psksecret <password>
    next
    edit "vpn_dc1-2"
        set interface "vlan-3001"
        set peertype any
        set proposal aes256-sha256
        set dhgrp 5
        set remote-gw 172.20.11.10
        set psksecret <password>
    next
end

Create two Phase 2 configurations, one for each data center. These configurations include:

  • Disabling pfs and replay
  • Enabling auto-negotiate to ensure VPN establishment
 
config vpn ipsec phase2-interface
    edit "vpn_dc1-1_p2"
        set phase1name "vpn_dc1-1"
        set proposal aes256-sha256
        set pfs disable
        set replay disable
        set auto-negotiate enable
    next
    edit "vpn_dc1-2_p2"
        set phase1name "vpn_dc1-2"
        set proposal aes256-sha256
        set pfs disable
        set replay disable
        set auto-negotiate enable
    next
end

2. Adding IP addresses to the tunnel interfaces

To establish the point-to-multipoint IPsec VPN between the branch and the data center, the tunnel interfaces must include the following IP addresses.

The IPsec VPN Interface configuration includes:

  • Setting ip to the local IP address of the VPN interface
  • Setting remote-ip to the data center FortiGate’s IPsec VPN interface IP address
 
config system interface
    edit "vpn_dc1-1"
        set vdom "root"
        set ip 10.254.0.2 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.254.0.1
        set interface "wan1"
    next
    edit "vpn_dc1-2"
        set vdom "root"
        set ip 10.254.1.2 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.254.1.1
        set interface "wan2"
    next
end

3. Implementing route discovery with BGP

BGP allows the branch and data center FortiGates to dynamically discover routes from each other. To make this happen add the data center FortiGate IPsec VPN tunnel interface IP addresses to the branch BGP configuration as  BGP peers.

Routes that have the same network mask, administrative distance, and priority are automatically considered for SD-WAN when the interfaces where those routes are learned are added to the SD-WAN interface group.

 

The branch BGP configuration includes:

  • Enabling ebgp-multipath
  • Enabling soft-reconfiguration and link-down-failover for each BGP peer
  • Adding the data center remote-as (which is 65500) to each peer configuration
  • Setting the prefix for the neighbor range to the network matching the BGP peers

To facilitate the fastest route failovers, the following timers are set to their lowest values:

  • scan-time
  • advertisement-interval
  • keep-alive timer
  • holdtime-timer
 
config router bgp
    set as 65501
    set router-id 10.254.0.2
    set keepalive-timer 1
    set holdtime-timer 3
    set ebgp-multipath enable
    set scan-time 5
    set distance-external 1
    config neighbor
        edit "10.254.0.1"
            set advertisement-interval 1
            set link-down-failover enable
            set soft-reconfiguration enable
            set remote-as 65500
        next
        edit "10.254.1.1"
            set advertisement-interval 1
            set link-down-failover enable
            set soft-reconfiguration enable
            set remote-as 65500
        next
    end
end

4. Setting up the load balancing SD-WAN configuration

The SD-WAN configuration sets up load balancing based on link quality. Link quality is determined by health checking; which measures jitter, packet loss, and latency on each link. FortiOS dynamically creates policy routes that send traffic over the link with the highest quality.

Create an SD-WAN Interface (also called a virtual WAN link) and add the IPsec VPN tunnel interfaces to it. These members are also the BGP neighbors that are tied to specific interfaces.  
config system virtual-wan-link
    set status enable
    config members
        edit 1
            set interface "vpn_dc1-1"
        next
        edit 2
            set interface "vpn_dc1-2"
        next
    end
end
 

Create SD-WAN Health-Checks for each data center network. Set server to the IP address of a server on the data center network.

 
config system virtual-wan-link
    config health-check
        edit "datacenter1-net"
            set server "10.200.1.1"
            set interval 1
            set failtime 1
            set recoverytime 3
        next
        edit "datacenter2-net"
             set server "10.200.2.1"
             set interval 1
             set failtime 1
             set recoverytime 3
         end
     end

Add SD-WAN Service Rules to define the criteria for the policy routes. Criteria include:

  • Protocol
  • Destination Address
  • Source Address
  • Identity Based Group
  • Internet Service Definition
  • Source Port
  • Destination Port
  • Destination Tag

To dynamically determine the networks the policy routes point to, the routes learned from a BGP neighbor are matched against a route map and matching routes are tagged. The service rules determine the routes to use based on these tags.

 
config system virtual-wan-link
    config service
        edit 1
            set mode priority
            set dst-tag 10
            set health-check "datacenter1-net"
            set priority-members 1 2
        next
        edit 2
             set mode priority
             set dst-tag 10
             set health-check "datacenter2-net"
             set priority-members 1 2
         next
    end
end

5. Controlling access from branch networks

Create firewall policies to allow users on the branch office networks to access the data center networks. Security profiles can be enabled on these firewall policies to inspect layer 7 traffic.

Policies to Allow traffic from the branch office to the data center networks:

  • Source interface: Internal interface
  • Destination interface: SD-WAN interface
  • Source Address: Branch networks
  • Destination Address: Data center networks
  • Action: Accept
  • Schedule: Always (or define a more restrictive schedule)
  • Service: Allowed services

Policies to allow traffic from the data center to the branch networks:

  • Source interface: SD-WAN interface
  • Destination interface: Internal interface
  • Source Address: Data center networks
  • Destination Address: Branch networks
  • Action: Accept
  • Schedule: Always (or define a more restrictive schedule)
  • Service: Allowed Services

Bill Dickie

Our Fearless Documentation Leader at Fortinet
After completing a science degree at the University of Waterloo, Bill began his professional life teaching college chemistry in Corner Brook, Newfoundland and fell into technical writing after moving to Ottawa in the mid '80s. Tech writing stints at all sorts of companies finally led to joining Fortinet to write the first FortiGate-300 Administration Guide.
  • Was this helpful?
  • Yes   No
  • Celio di Cavalcanti

    How to implement communication between SPOKES with ADVPN + SDWAN because the interface is dynamic.

  • Brian Holmes

    Could I use GRE over an existing MPLS network to replace 1 of the two Internet VPN’s?

    • bdickie

      GRE with SD-WAN is currently not supported.

  • Laura

    Hello. What if I want to have my two wan interface into the SD-WAN and create the IPSec tunnel that are not going to be part of SD-WAN? Also I’ll use static route. Is there a cook book for this implementation?

    • bdickie

      This configuration may be possible but we have not had the opportunity to test and validate it.

      • Laura

        Thanks I’m testing it this week in my lab. If it works, I’ll be sharing the configuration. Thanks again

        • bdickie

          Yes,it would be great if you could share your configuration with us.

  • Masud

    Hi Bill Dickie,
    Appreciate if you can post this articles with OSPF and minimum 2/3 branches. We want to check if it is feasible in our banking environment.

    • bdickie

      Hello, OSPF is not currently supported for SD-WAN. Just static routes and BGP.

  • Gary Swick

    Can this also be done with OSPF instead of BGP? We want to have two hubs (each with dual WAN connections) and then multiple spokes (each also with dual WAN connections)

    • bdickie

      Hello, the FortiOS SD-WAN feature does not currently OSPF. Only static routes and BGP are supported.