High Availability with FGCP (Expert)

Facebooktwittergoogle_plusredditpinterestlinkedinFacebooktwittergoogle_plusredditpinterestlinkedin

This recipe describes how to enhance the reliability of a network protected by a FortiGate unit by adding a second FortiGate unit and setting up a FortiGate Clustering Protocol (FGCP) High Availability cluster.

The FortiGate already on the network will be configured to become the primary unit by increasing its device priority and enabling override. The new FortiGate will be prepared by setting it to factory defaults to wipe any configuration changes. Then it will be licensed, configured for HA, and then connected to the FortiGate already on the network. The new FortiGate becomes the backup unit and its configuration is overwritten by the primary unit.

The recipe contains instructions for both the GUI and the CLI, with some parts of the configuration requiring use of the CLI. A simplified HA recipe that only requires use of the GUI is available here.

Before you start the FortiGates should be running the same FortiOS firmware version and interfaces should not be configured to get their addresses from DHCP or PPPoE.

1. Configuring the primary FortiGate

If the FortiGates in the cluster will be running FortiOS Carrier, apply the FortiOS Carrier license before configuring the cluster (and before applying other licenses). Applying the FortiOS Carrier license sets the configuration to factory defaults, requiring you to repeat steps performed before applying the license.

Connect to the primary FortiGate and go to System > Dashboard > Status and locate the System Information widget.

Change the unit’s Host Name to identify it as the primary FortiGate.

 

You can also enter this CLI command:

 config system global
    set hostname Primary_FortiGate
 end

If you have not already done so, register the primary FortiGate and apply licenses to it before setting up the cluster. This includes FortiCloud activation and FortiClient licensing, and entering a license key if you purchased more than 10 Virtual Domains (VDOMs). All FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient and VDOMs. You can also install any third-party certificates on the primary FortiGate before forming the cluster. Once the cluster is formed third-party certificates are synchronized to the backup FortiGate. FortiToken licenses can be added at any time because they are synchronized to all of the workers.

Enter this CLI command to set the HA mode to active-passive, set a group name and password, increase the device priority to a higher value (for example, 250) and enable override.

 config system ha
    set mode a-p
    set group-id 25

    set group-name My-HA-Cluster
    set password
    set priority 250
    set override enable
    set hbdev ha1 50 ha2 50
 end

Enabling override and increasing the device priority means this unit should always become the primary unit.

This command also selects ha1 and ha2 to be the heartbeat interfaces and sets their priorities to 50.

If you have more than one cluster on the same network, each cluster should have a different group id. Changing the group id changes the cluster interface virtual MAC addresses. If your group id setting causes MAC address conflict you can select a different group id.

You can also use the GUI to configure most of these settings.

 

Override  and the group id can only be configured from the CLI. 

config system ha
    set group-id 25

    set override enable
end

The FortiGate unit negotiates to establish an HA cluster. When you select OK you may temporarily lose connectivity with the FortiGate unit as FGCP negotiation takes place and the MAC addresses of the FortiGate unit are changed to HA virtual MAC addresses. These virtual MAC addresses are used for failover. The actual virtual MAC address assigned to each FortiGate interface depends on the HA group ID. Since this example does not involved changing the HA group ID, the FortiGate unit’s interfaces will have the following MAC addresses: 00:09:0f:09:00:00, 00:09:0f:09:00:01, 00:09:0f:09:00:02 and so on.

To reconnect sooner, you can update the ARP table of your management PC by deleting the ARP table entry for the FortiGate unit (or just deleting all arp table entries). You can usually delete the arp table from a command prompt using a command similar to arp -d.

To confirm these MAC address changes, you can use the get hardware nic (or diagnose hardware deviceinfo nic) command to view the virtual MAC address of any FortiGate unit interface. Depending on the FortiGate model, the output from this command could include lines similar to the following:

Current_HWaddr:   00:09:0f:09:00:00
Permanent_HWaddr  02:09:0f:78:18:c9

2. Configuring the backup FortiGate

Enter this command to reset the new FortiGate to factory default settings.

execute factoryreset

You can skip this step if the new FortiGate is fresh from the factory. But if its configuration has been changed at all it is recommended to set it back to factory defaults to reduce the chance of synchronization problems.

Change the firmware running on the new FortiGate to be the same version as is running on the primary unit.

Register the backup FortiGate and apply licenses to it before setting up the cluster. This includes FortiCloud activation, FortiClient and FortiToken licensing, and entering a license key if you purchased more than 10 Virtual Domains (VDOMs). All FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient and VDOMs.

Go to System > Dashboard > Status and change the unit’s Host Name to identify it as the backup FortiGate.

  

You can also enter this CLI command:

config system global
    set hostname Backup_FortiGate
 end

Duplicate the primary unit HA settings, except set the Device Priority to a lower value (for example, 50) and do not enable override.

config system ha
    set mode a-p
    set group-id 25

    set group-name My-HA-Cluster
    set password
    set priority 50
    set hbdev ha1 50 ha2 50
 end

3. Connecting the cluster

Connect the HA cluster as shown in the initial diagram. Making these connections will disrupt network traffic as you disconnect and re-connect cables.

When connected the primary and backup FortiGates find each other and negotiate to form an HA cluster.  The Primary unit synchronizes its configuration with the backup FortiGate. Forming the cluster happens automatically with minimal or no disruption to network traffic.

4. Checking cluster operation and disabling override

Check the cluster synchronization status to make sure the primary and backup units have the same configuration. Log into the primary unit CLI and enter this command:

diag sys ha cluster-csum

The command output lists all members’ checksums. If both cluster units have identical checksums you can be sure that their configurations are synchronized. If the checksums are different wait a short while and enter the command again. Repeat until the checksums are identical. It may take a while for some parts of the configuration to be synchronized. If the checksums never become identical contact Fortinet support to help troubleshoot the problem.

When the checksums are identical, disable override on the primary unit (recommended).

config system ha
    set override disable
end

The HA cluster dynamically responds to network conditions. If you keep override enabled the same FortiGate will always be the primary FortiGate. Because of this, however; the cluster may negotiate more often potentially disrupting traffic.

If you disable override it is more likely that the new FortiGate unit could become the primary unit. Disabling override is recommended unless its important that the same FortiGate remains the primary unit.

Connect to the primary FortiGate GUI  and go to System > Config > HA to view the cluster information.

 
Select View HA Statistics for more information on how the cluster is operating and processing traffic.  

5. Results

Normally, traffic should now be flowing through the primary FortiGate. However, if the primary FortiGate is unavailable, traffic should failover and the backup FortiGate will be used. Failover will also cause the primary and backup FortiGates to reverse roles, even when both FortiGates are available again.
To test this, ping the IP address 8.8.8.8 using a PC on the internal network. After a moment, power off the primary FortiGate. You will see a momentary pause in the Ping results, until traffic diverts to the backup FortiGate, allowing the Ping traffic to continue.

For further reading, check out Configuring and connecting HA clusters in the FortiOS 5.2 Handbook.

Bill Dickie

Our Fearless Documentation Leader at Fortinet
After completing a science degree at the University of Waterloo, Bill began his professional life teaching college chemistry in Corner Brook, Newfoundland and fell into technical writing after moving to Ottawa in the mid '80s. Tech writing stints at all sorts of companies finally led to joining Fortinet to write the first FortiGate-300 Administration Guide.
  • Was this helpful?
  • Yes   No
If these steps don’t start HA mode, make sure that none of the FortiGate’s interfaces use DHCP or PPPoE addressing.
If these steps don’t start HA mode, make sure that none of the FortiGate’s interfaces use DHCP or PPPoE addressing.
If you are using port monitoring, you can also unplug the primary FortiGate’s Internet-facing interface to test failover.
  • Mark A. Tiberio

    Can anyone go into more detail on the switching? We have a core switch for our internal network. Do I need a switch between the fortigates and my core? also i have 2 isps, does this require 4 switches total?

    • bdickie

      You can use fewer switches yes as long as the internal and external traffic is separated and the switch can respond to the gratuitous arps when a failover occurs.

      You can connect the internal interface of both FortiGates to the same core switch yes. This section of the HA guide has some information about different switch configurations. http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-high-availability-52/HA_third-party.htm.

      If your FortiGates have dual internet connections then yes you can add 4 switches or you can use fewer as long as they are configured correctly.

  • gustavo zarini

    I have this implementation with 3 lacp for each fgt. Each fgt connects to a cisco 2960xr stacked.
    I cannot implement active/active cluster since internet access is failing. but only fails witch polocy that has any security profile applied. if i don’t apply filters internet access works fine.

  • Bjørn Tore

    I use HA in A/P and BGP with BFD to my PE-routers. What is the BCP on this setup? Should I rely purely on BFD to trigger a failover, and will it in fact trigger a fail-over? Or should I use link-monitor? And if the latter – which server should I use? If I use my primary next-hop, it will be unavailable as long as I am on my slave – and then the fail-over to preferred master will never happen.

  • Anderson Porfirio

    You forget, that we need remove DHCP or PPoE from all interfaces to set the ha mode diferente from standalone.

    Anderson Porfirio

    • bdickie

      Thanks for your comment and you are correct. I will update the recipe to make this clearer.

  • Ed

    If my A/A cluster consists of aggregate (LACP) ports (Full Mesh) up-linking to a 2xCisco Switch Stack/Nexus vPC, should the Cisco side be configured as one ether-channel or as two separate ether-channels (one for each Fortigate unit)?

    I don’t fully understand this part “LACP, 802.3ad aggregation and third-party switches”
    http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-high-availability-52/HA_third-party.htm

    • bdickie

      I am not sure of the answer to this question. I will see what I can find out and get back to you. Feel free to contact Fortinet Support about this instead of waiting for us to reply. It would be great if this all resulted in improvements to the HA guide.

    • bdickie

      The Cisco switch should be configured with two separate Ethernet channels, one for each FortiGate unit. We will improve the explanation in the section you reference, but the idea is that if your switch supports multiple LAG groups you need to configure them to be able to support A-A HA. If your switch can’t support multiple LAG groups then it can’t support A-A HA and you need to enable lacp-ha-slave just to support A-P HA.

      • Ed

        Thank you for the answer and explanation

  • Ed

    FortiOS documentation is silent on the IP addressing done on the slave unit for the wan and internal interfaces.
    Should the interfaces be left blank without IPs awaiting config sync from the master?
    Once the config is sync’ed, what IP does the slave unit use in an Actiev/Active cluster setup?

    • bdickie

      There is no special requirement for the backup (or slave) unit’s IP addresses. They can be anything. As long as the backup and the primary can synchronize over the heartbeat link the cluster will form. After the cluster forms the backup unit will be configured with the same IP addresses as the primary unit. You can very this from the CLI by using the execute ha manage command to log into the backup unit’s CLI and using the get system interface command to view the backup unit’s IP addresses. Feel free to reply if you need more information.

      • Ed

        Is the below correct;
        – The management interface IP is also overwritten on the slave unit.
        – The heart beat interfaces don’t need IP addresses.

        • bdickie

          The idea of the reserved management interface is you can configure a different IP address for a selected interface on the primary and backup units. This reserved management interface IP address is not synchronized (or overwritten). (see http://help.fortinet.com/fos50hlp/52data/Content/FortiOS/fortigate-high-availability-52/HA_maintenance.htm (and scroll down to the “Managing individual cluster units using a reserved management interface” section).

          Yes, the HA heartbeat uses hidden IP addresses. You can display them using the get system ha status command (see http://help.fortinet.com/fos50hlp/52data/Content/FortiOS/fortigate-high-availability-52/HA_failover.htm and scroll down to the “HA Heartbeat interface IP addresses” section. You can configure IP addresses on the heartbeat interfaces if you want to use them for network traffic (but this is not recommended because the heartbeat can use a lot of bandwidth). But the HA heartbeat does not use these IP addresses.

          • Ed

            Much appreciated. Now I fully understand.

          • Ed

            Having come from a Cisco ASA/Checkpoint background, i just couldn’t add up things on the Fortigate HA config.

  • Lord Amarant

    can use the config for a geographical cluster over vpls?

  • Pierpaolo Panarotto

    Could i use the config for a geographical cluster over vpls?

  • Dunwright

    Would the config for Active/Active HA be the same except for setting the mode to a-a?

    • bdickie

      Yes, the active/active configuration is the same except for selecting the HA mode.