High Availability with FGCP (Expert)

Facebooktwittergoogle_plusredditpinterestlinkedinFacebooktwittergoogle_plusredditpinterestlinkedin

This recipe describes how to enhance the reliability of a network protected by a FortiGate unit by adding a second FortiGate unit and setting up a FortiGate Clustering Protocol (FGCP) High Availability cluster.

The FortiGate already on the network will be configured to become the primary unit by:

  • Licensing it (if required)
  • Enabling HA
  • Increasing its device priority
  • Enabling override

The new FortiGate will be prepared by:

  • Setting it to factory defaults to wipe any configuration changes
  • Licensing it (if required)
  • Enabling HA without changing the device priority and override
  • Connecting it to the FortiGate already on the network.

The new FortiGate becomes the backup unit and its configuration is overwritten by the primary unit.

The recipe contains instructions for both the GUI and the CLI, with some parts of the configuration requiring use of the CLI. A simplified HA recipe that only requires use of the GUI is available here.

Before you start, the FortiGates should be running the same FortiOS firmware version and interfaces should not be configured to get their addresses from DHCP or PPPoE.

Find this recipe for other FortiOS versions
5.2 | 5.4 | 5.6

1. Configuring the primary FortiGate

Connect to the primary FortiGate and locate the System Information Dashboard widget. Click on the System Information dashboard widget and select Configure settings in System > Settings.

Change the unit’s Host name to identify it as the primary FortiGate.

You can also enter this CLI command:

config system global
  set hostname External-Primary
end
Register and apply licenses to the primary FortiGate before configuring it for HA operation. This includes licensing for FortiCare Support, IPS, AntiVirus, Web Filtering, Mobile Malware, FortiClient, FortiCloud, and additional virtual domains (VDOMs). All FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient, and VDOMs. FortiToken licenses can be added at any time because they are synchronized to all cluster members.
You can also install any third-party certificates on the primary FortiGate before forming the cluster. Once the cluster is formed, third-party certificates are synchronized to the backup FortiGate.

Enter this CLI command to set the HA mode to active-passive, set a group id, group name and password, increase the device priority to a higher value (for example, 250) and enable override.

config system ha
  set mode a-p
  set group-id 25
  set group-name External-HA-Cluster
  set password
  set priority 250
  set override enable
  set hbdev port3 200 port4 100
end

Enabling override and increasing the device priority means this unit should always become the primary unit.

This command also selects port3 and port4 to be the heartbeat interfaces and sets their priorities to 200 and 100 respectively. Its a best practice to set different priorities for the heartbeat interfaces (but not a requirement).

If you have more than one cluster on the same network, each cluster should have a different group id. Changing the group id changes the cluster interface virtual MAC addresses. If your group id setting causes MAC address conflict you can select a different group id.

You can also use the GUI (System > HA) to configure most of these settings.

Override and the group id can only be configured from the CLI.
config system ha
  set group-id 25
  set override enable
end

The FortiGate unit negotiates to establish an HA cluster. You may temporarily lose connectivity with the FortiGate unit as FGCP negotiation takes place and the MAC addresses of the FortiGate interfaces are changed to HA virtual MAC addresses. These virtual MAC addresses are used for failover. The actual virtual MAC address assigned to each FortiGate interface depends on the HA group ID. Since this example does not involve changing the HA group ID, the FortiGate unit’s interfaces will have the following MAC addresses: 00:09:0f:09:00:00, 00:09:0f:09:00:01, 00:09:0f:09:00:02 and so on.

To reconnect sooner, you can update the ARP table of your management PC by deleting the ARP table entry for the FortiGate unit (or just deleting all ARP table entries). You can usually delete the ARP table from a command prompt using a command similar to arp -d.

To confirm these MAC address changes, you can use the get hardware nic (or diagnose hardware deviceinfo nic) command to view the virtual MAC address of any FortiGate unit interface. Depending on the FortiGate model, the output from this command could include lines similar to the following:

Current_HWaddr:   00:09:0f:09:00:00
Permanent_HWaddr  02:09:0f:78:18:c9

2. Configuring the backup FortiGate

Enter this command to reset the new FortiGate that will become the backup FortiGate to factory default settings.

execute factoryreset

You can skip this step if the new FortiGate is fresh from the factory. But if its configuration has been changed at all it is recommended to set it back to factory defaults to reduce the chance of synchronization problems.

If required, change the firmware running on the new FortiGate to be the same version as is running on the primary unit.

Register and apply licenses to the new FortiGate unit before adding it to the HA cluster. This includes licensing for FortiCare Support, IPS, AntiVirus, Web Filtering, Mobile Malware, FortiClient, FortiCloud, and additional virtual domains (VDOMs). All FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient, and VDOMs. FortiToken licenses can be added at any time because they are synchronized to all cluster members.

Click on the System Information dashboard widget and select Configure settings in System > Settings. Change the FortiGate’s Host name to identify it as the backup FortiGate.

You can also enter this CLI command:
config system global
  set hostname External-Backup
end

Duplicate the primary unit HA settings, except set the Device Priority to a lower value (for example, 50) and do not enable override.

config system ha
  set mode a-p
  set group-id 25
  set group-name External-HA-Cluster
  set password
  set priority 50
  set hbdev port3 200 port4 100
end

3. Connecting the cluster

Connect the HA cluster as shown in the network diagram. Making these connections will disrupt network traffic as you disconnect and re-connect cables.

If possible, make direct Ethernet connections between the heartbeat interfaces of the two FortiGate units.

Switches must be used between the cluster and the Internet and between the cluster and the internal networks as shown in the network diagram. You can use any good quality switches to make these connections. You can also use one switch for all of these connections as long as you configure the switch to separate traffic from the different networks.

When connected, the primary and backup FortiGates find each other and negotiate to form an HA cluster. The Primary unit synchronizes its configuration with the backup FortiGate. Forming the cluster happens automatically with minimal or no disruption to network traffic.

4. Checking cluster operation and disabling override

Check the cluster synchronization status to make sure the primary and backup units have the same configuration. Log into the primary unit CLI and enter this command:

diagnose sys ha checksum cluster

The command output lists all cluster members’ configuration checksums. If both cluster units have identical checksums you can be sure that their configurations are synchronized. If the checksums are different, wait a short while and enter the command again. Repeat until the checksums are identical. It may take a while for some parts of the configuration to be synchronized. If the checksums never become identical contact Fortinet Support to help troubleshoot the problem.

The HA Status Dashboard widget also shows if the cluster units are synchronized. Mouse over each FortiGate in the cluster to verify that they both have the same checksum.

When the checksums are identical, disable override on the primary unit (recommended).

config system ha
   set override disable
end

The HA cluster dynamically responds to network conditions. If you keep override enabled, the same FortiGate will always be the primary FortiGate. Because of this, however; the cluster may negotiate more often potentially increasing traffic disruptions.

If you disable override it is more likely that the new FortiGate unit could become the primary unit. Disabling override is recommended unless its important that the same FortiGate remains the primary unit.

From the HA Status widget, select Configure Settings in System > HA (or go to System > HA) to view the cluster status.

From the HA Status widget you can also select Show HA Historical Events to see the most recent HA system status messages.

5. Results

Normally, traffic should now be flowing through the primary FortiGate. If the primary FortiGate is unavailable traffic fails over to the backup FortiGate. Failover also causes the primary and backup FortiGates to reverse roles, even when both FortiGates are available again.
To test this, ping the IP address 8.8.8.8 using a PC on the internal network. After a moment, power off the primary FortiGate. You will see a momentary pause in the ping results, until traffic diverts to the backup FortiGate, allowing the ping traffic to continue.

For further reading, check out FGCP configuration examples and troubleshooting in the FortiOS 5.6 Handbook.

Bill Dickie

Our Fearless Documentation Leader at Fortinet
After completing a science degree at the University of Waterloo, Bill began his professional life teaching college chemistry in Corner Brook, Newfoundland and fell into technical writing after moving to Ottawa in the mid '80s. Tech writing stints at all sorts of companies finally led to joining Fortinet to write the first FortiGate-300 Administration Guide.
  • Was this helpful?
  • Yes   No
Also, you cannot use a switch port as an HA heartbeat interface. If necessary, convert the switch port to individual interfaces (see Choosing your FortiGate’s switch mode).
If the FortiGates in the cluster will be running FortiOS Carrier, apply the FortiOS Carrier license before configuring the cluster (and before applying other licenses). Applying the FortiOS Carrier license sets the configuration to factory defaults, requiring you to repeat steps performed before applying the license.
If these steps don’t start HA mode, make sure that none of the FortiGate’s interfaces use DHCP or PPPoE addressing.
If the FortiGates in the cluster will be running FortiOS Carrier, apply the FortiOS Carrier license before configuring the cluster (and before applying other licenses). Applying the FortiOS Carrier license sets the configuration to factory defaults, requiring you to repeat steps performed before applying the license.
If these steps don’t start HA mode, make sure that none of the FortiGate’s interfaces use DHCP or PPPoE addressing.
This example uses two port3 and port4, but you can use any interfaces for HA heartbeat interfaces. A best practice is to use interfaces that do not process traffic, but this is not a requirement.
If you are using port monitoring, you can also unplug the primary FortiGate’s Internet-facing interface to test failover.