FGCP High Availability Troubleshooting

Facebooktwittergoogle_plusredditpinterestlinkedinFacebooktwittergoogle_plusredditpinterestlinkedin

This post is intended to help you find and fix some common and not so common FortiGate Clustering Protocol (FGCP) HA problems.

1. Before you set up a cluster

Before you set up an FortiGate FGCP cluster ask yourself these questions:

  • Do all the FortiGates have the same hardware version and the same hardware configuration?
  • Do all the FortiGates have the same firmware build?
  • Are all the FortiGates set to the same operating mode (NAT or Transparent)?
  • Are all the FortiGates operating in single VDOM mode?
  • If the FortiGates are operating in multiple VDOM mode do they all have the same VDOM configuration?

In some cases you may be able to form a cluster if your FortiGates have different firmware builds, different VDOM configurations, and are in different operating modes. However, if you encounter problems when forming a cluster you may be able to resolve them by installing the same firmware build on each unit and giving them the same VDOM configuration and operating mode. If possible you could also reset them all to factory defaults and start over.

If the FortiGates in the cluster have different licenses, the cluster will form but it will operate with the lowest licensing level.

2. Troubleshooting Licensing

All of the FortiGates in a cluster must have the same level of licensing. This includes FortiGuard, FortiCloud, FortiClient, VDOMs (if applicable) and FortiOS Carrier (if applicable).

If one of the FortiGates in a cluster has a lower level of licensing than other FortiGates in the cluster, then all of the FortiGates in the cluster will revert to that lower licensing level. For example, if you only purchase FortiGuard Web Filtering for one of the FortiGates in a cluster, when the cluster is operating, none of the cluster units will support FortiGuard Web Filtering.

An exception is FortiToken licensing. FortiToken activations are completed one FortiGate unit and synchronized to all of the FortiGates in the cluster.

3. Troubleshooting hardware revisions

Many FortiGate platforms have gone through multiple hardware versions and in some cases the hardware changes may prevent cluster formation. If you run into this problem you can use the following command on each FortiGate in the cluster to cause the cluster to ignore different hardware versions:

execute ha ignore-hardware-revision enable

This command is only available on FortiGates that have had multiple hardware revisions. So if the command isn’t present then hardware version issues should not prevent cluster formation.

By default the command is set to prevent cluster formation between FortiGates with different hardware revisions. You can enter the following command to view its status:

execute ha ignore-hardware-revision status

Usually the incompatibility is caused by different hardware versions having different hard disks and enabling this command disables each FortiGate’s hard disks. As a result of disabling hard disks the cluster will not support logging to the hard disk or WAN Optimization.

If the FortiGates do have compatible hardware versions or if you want to run a FortiGate in standalone mode you can enter the following command to disable ignoring the hardware revision and enable the hard disks:

execute ha ignore-hardware-revision disable

Affected models include but are not limited to:

  • FortiGate-100D
  • FortiGate-300C
  • FortiGate-600C
  • FortiGate-800C
  • FortiGate-80C and FortiWiFi-80C
  • FortiGate-60C

4. Troubleshooting the initial cluster configuration

This step describes how to check a cluster when it first starts up to make sure that it is configured and operating correctly. This section assumes you have already configured your HA cluster and it appears to be up and running normally.

To verify that a cluster can process traffic and react to a failure:

  1. Add a basic security policy configuration and send network traffic through the cluster to confirm connectivity. For example, if the cluster is installed between the Internet and an internal network, set up a basic internal to external security policy that accepts all traffic. Then from a PC on the internal network, browse to a website on the Internet or ping a server on the Internet to confirm connectivity.
  2. From your management PC, set ping to continuously ping through the cluster, and then start a large download, or in some other way establish ongoing traffic through the cluster.
  3. While traffic is going through the cluster, disconnect the power from one of the cluster units. You could also shut down or restart a cluster unit. Traffic should continue with minimal interruption.
  4. Start up or reconnect the cluster unit that you disconnected. The FortiGate should re-join the cluster with little or no affect on traffic.
  5. Disconnect a cable from one of the HA heartbeat interfaces. The cluster should keep functioning, using the other HA heartbeat interface.
  6. If you have port monitoring enabled, disconnect a network cable from a monitored interface. Traffic should continue with minimal interruption.
  

5. Verifying the cluster configuration from the GUI

If a cluster is formed you can do the following to verify its status and configuration.

 

Log into the cluster GUI. Verify that the System Information widget lists all of the cluster units.

 1-trbl-system-information
Check the Unit Operation widget to verify that the correct cluster unit interfaces are connected.  2-trbl-unit-operation
Go to System > HA or on the System Information dashboard widget select HA Status > Configure and verify that all of the cluster units are displayed on the  HA Cluster list.  3-trbl-cluster-list
From the cluster members list, edit the primary unit (master) and verify the cluster configuration.  4-trbl-ha-config

6. Troubleshooting the cluster configuration from the GUI

Try this if the FortiGates don’t successfully form a cluster:

Connect to each cluster unit GUI and verify that the HA configurations are the same. The HA configurations of all cluster units must be identical. Even though the HA configuration is very simple you can easily make a small mistake that prevents a FortiGate from joining a cluster. (I speak form personal experience here.)

If the configurations are the same, try re-entering the HA password on each cluster unit in case you made an error typing the password when configuring one of the cluster units.

Check that the correct interfaces of each cluster unit are connected. Check the cables and interface LEDs. Use the Unit Operation dashboard widget, system network interface list, or cluster members list to verify that each interface that should be connected actually is connected. If a link is down re-verify the physical connection. Try replacing network cables or switches as required.

  

7. Verifying the cluster configuration from the CLI

If a cluster is formed you can do the following to verify its status and configuration.

Log into each cluster unit CLI. You can use the GUI CLI console, SSH, or a direct console port connection.

Enter the command get system status. Look for the current HA mode in the command output. If the cluster is operating correctly and you have connected to the primary unit you should see something like this:

Current HA mode: a-a, master

You can connect to the backup or subordinate unit using a console port or by connecting to the console CLI and using the execute ha manage command to connect to the backup unit. If the cluster is operating correctly you will see something like this:

Current HA mode: a-a, backup

If the FortiGate is not operating in HA mode the get system status command output is something like this:

Current HA mode: standalone

Verify that the get system ha status command displays all of the cluster units. For example, for a cluster of three FortiGate units, the command output should contain something like this:

Master: 5001d-slot3     , FG-5KD3914800344
Slave : 5001d-slot5     , FG-5KD3914800353
Slave : 5001d-slot4     , FG-5KD3914800284

Enter the get system ha command to verify that the HA configuration is correct and the same for each cluster unit.

get system ha
group-id : 0
group-name : External-HA-cluster
mode : a-p
password : *
hbdev : "port3" 50 "port4" 50
.
.
.

8. Troubleshooting the cluster configuration from the CLI

If the FortiGates don’t successfully form a cluster, try using the following command to re-enter the cluster password.  Do this for each cluster unit in case you made an error typing the password when configuring one of the cluster units.

config system ha
    set password <password>
end

Check that the correct interfaces of each cluster unit are connected. Check the cables and interface LEDs. Use the get hardware nic <interface_name> command to confirm that each interface is connected. If the interface is connected the command output should contain a Link: up entry similar to the following:

get hardware nic port1
.
.
.
Link: up
.
.
.

If the link is down, re-verify the physical connection. Try replacing network cables or switches as required.

More troubleshooting information

Much of the information in the  HA guide can be useful for troubleshooting HA clusters. Here are some links to sections with more information.

  • If sessions are lost after a failover you may need to change route-ttl to keep synchronized routes active longer. See Synchronizing kernel routing tables.
  • In rare cases, sometimes after a cluster unit has been replaced it is possible that a cluster will not form because the disk partition sizes of the cluster units are different. You can use the following command to check the disk storage checksum of each cluster unit. If the checksums are different then contact Fortinet support for help in setting up compatible storage partitions.
diagnose sys ha showcsum 1 system | grep storage
  • To control which cluster unit becomes the primary unit, you can change the device priority and enable override. See Controlling primary unit selection using device priority and override.
  • Changes made to a cluster can be lost if override is enabled. See Configuration changes can be lost if override is enabled.
  • When override is enabled, after a failover traffic may be disrupted if the primary unit rejoins the cluster before the session tables are synchronized or for other reasons such as if the primary unit is configured for DHCP or PPPoE. See Delaying how quickly the primary unit rejoins the cluster when override is enabled.
  • In some cases, age differences among cluster units result in the wrong cluster unit becoming the primary unit. For example, if a cluster unit set to a high priority reboots, that unit will have a lower age than other cluster units. You can resolve this problem by resetting the age of one or more cluster units. See Primary unit selection and age. You can also adjust how sensitive the cluster is to age differences. This can be useful if large age differences cause problems. See Cluster age difference margin (grace period) and Changing the cluster age difference margin.
  • If one of the cluster units needs to be serviced or removed from the cluster for other reasons, you can do so without affecting the operation of the cluster. See Disconnecting a cluster unit from a cluster.
  • The web-based manager and CLI will not allow you to configure HA if you have enabled FGSP HA. See FortiGate Session Life Support Protocol (FGSP).
  • The GUI and CLI will not allow you to configure HA if one or more FortiGate unit interfaces is configured as a PPTP or L2TP client.
  • The FGCP is compatible with DHCP and PPPoE but care should be taken when configuring a cluster that includes a FortiGate interface configured to get its IP address with DHCP or PPPoE. Fortinet recommends that you turn on DHCP or PPPoE addressing for an interface after the cluster has been configured. See FortiGate HA compatibility with DHCP and PPPoE.
  • Some third-party network equipment may prevent HA heartbeat communication, resulting in a failure of the cluster or the creation of a split brain scenario. For example, some switches use packets with the same Ethertype as HA heartbeat packets use for internal functions and when used for HA heartbeat communication the switch generates CRC errors and the packets are not forwarded. See Heartbeat packet Ethertypes.
  • Very busy clusters may not be able to send HA heartbeat packets quickly enough, also resulting in a split brain scenario. You may be able to resolve this problem by modifying HA heartbeat timing. See Modifying heartbeat timing.
  • Very busy clusters may suffer performance reductions if session pickup is enabled. If possible you can disable this feature to improve performance. If you require session pickup for your cluster, several options are available for improving session pickup performance. See Improving session synchronization performance.
  • If it takes longer than expected for a cluster to failover you can try changing how the primary unit sends gratuitous ARP packets. See Changing how the primary unit sends gratuitous ARP packets after a failover on page 1.
  • You can also improve failover times by configuring the cluster for subsecond failover. See Subsecond failover and Failover performance on page 1.
  • When you first put a FortiGate unit in HA mode you may loose connectivity to the unit. This occurs because HA changes the MAC addresses of all FortiGate unit interfaces, including the one that you are connecting to. The cluster MAC addresses also change if you change some HA settings such as the cluster group ID. The connection will be restored in a short time as your network and PC updates to the new MAC address. To reconnect sooner, you can update the ARP table of your management PC by deleting the ARP table entry for the FortiGate unit (or just deleting all ARP table entries). You may be able to delete the arp table of your management PC from a command prompt using a command similar to arp -d.
  • Since HA changes all cluster unit MAC addresses, if your network uses MAC address filtering you may have to make configuration changes to account for the HA MAC addresses.
  • A network may experience packet loss when two FortiGate HA clusters have been deployed in the same broadcast domain. Deploying two HA clusters in the same broadcast domain can result in packet loss because of MAC address conflicts. The packet loss can be diagnosed by pinging from one cluster to the other or by pinging both of the clusters from a device within the broadcast domain. You can resolve the MAC address conflict by changing the HA Group ID configuration of the two clusters. The HA Group ID is sometimes also called the Cluster ID. See Diagnosing packet loss with two FortiGate HA clusters in the same broadcast domain.
  • The cluster CLI displays slave is not in sync messages if there is a synchronization problem between the primary unit and one or more subordinate units. See How to diagnose HA out of sync messages.
  • If you have configured dynamic routing and the new primary unit takes too long to update its routing table after a failover you can configure graceful restart and also optimize how routing updates are synchronized. See Configuring graceful restart for dynamic routing failover and Synchronizing kernel routing tables.
  • Some switches may not be able to detect that the primary unit has become a subordinate unit and will keep sending packets to the former primary unit. This can occur after a link failover if the switch does not detect the failure and does not clear its MAC forwarding table. See Updating MAC forwarding tables when a link failover occurs.
  • If a link not directly connected to a cluster unit (for example, between a switch connected to a cluster interface and the network) fails you can enable remote link failover to maintain communication. See Remote link failover.
  • If you find that some cluster units are not running the same firmware build you can reinstall the correct firmware build on the cluster to upgrade all cluster units to the same firmware build. See Synchronizing the firmware build running on a new cluster unit.

Bill Dickie

Our Fearless Documentation Leader at Fortinet
After completing a science degree at the University of Waterloo, Bill began his professional life teaching college chemistry in Corner Brook, Newfoundland and fell into technical writing after moving to Ottawa in the mid '80s. Tech writing stints at all sorts of companies finally led to joining Fortinet to write the first FortiGate-300 Administration Guide.

Latest posts by Bill Dickie (see all)

  • Was this helpful?
  • Yes   No