How to upgrade one unit in an HA cluster

Facebooktwittergoogle_pluslinkedinFacebooktwittergoogle_pluslinkedin

In this recipe, which starts with a FortiGate Clustering Protocol (FGCP) cluster of two FortiGate units, you will upgrade the primary unit’s firmware, while keeping the subordinate unit as a failsafe backup running the original firmware. 

If the new firmware upgrades and runs successfully, you can quickly upgrade the entire cluster to the new firmware. If the new firmware fails during or after the upgrade, you can quickly revert the cluster to the older firmware.

This recipe increases the effort needed to upgrade cluster firmware but allows easily falling back to the original firmware version and FortiGate configuration with minimal network interruption.

Normally when you upgrade a cluster, network traffic is not interrupted. However, upgrading one unit in a cluster results in minor network disruptions similar to upgrading the firmware of a single FortiGate unit.

This recipe requires you to enable the dedicated or reserved HA management interface feature.

This example uses the following interfaces:

  • Internal1 is the reserved management interface
  • Internal2 is connected to the Internal Network
  • Wan1 is connected to the Internet
  • Internal4 and Internal5 are the HA heartbeat interfaces

1. Enable the HA reserved management interface feature

You can configure the HA reserved management interface feature when originally setting up the cluster.

If the cluster is already running, log into the primary unit and go to  System > Config > HA, select the primary unit, enable the reserved management interface, and select an interface.

Then go to System > Network > Interface and configure the interface that you selected.

 

You can also use the following command to set up the reserved management interface from the CLI. This is also the only way to add a default gateway for the reserved management interface if one is required.

config system ha
   set ha-mgmt-status enable
   set ha-mgmt-interface internal1
   set ha-mgmt-interface-gateway 10.11.101.2
end

To configure the subordinate unit’s reserved management interface, from the primary unit CLI use the execute ha manage command to access the subordinate unit’s CLI. Then use the config system interface command to set the IP address for the subordinate unit reserved management interface. You can also use the set ha-mgmt-interface-gateway command to configure the default gateway.

Enabling and selecting the reserved management interface is synchronized to both cluster members. The management interface gateway and the configuration of the management interface is not synchronized.

2. Disable HA configuration synchronization

Enter this command to disable HA configuration synchronization. You can enter this command from any CLI prompt on the primary unit (master) or subordinate unit (slave). The change is synchronized to both FortiGate units in the cluster.

config system ha
   set sync-config disable
end

3. Back up the configuration of each cluster unit

Use the reserved management IP addresses to log into the GUI of each cluster unit and verify that the serial numbers and role of the unit in the cluster match. The first image shows an example primary unit (master) and the second an example subordinate unit (slave).

 Primary unit (master)

Subordinate unit (slave)

From the system information dashboard widget of each cluster unit GUI, back up each cluster unit’s configuration. Back up both configurations since some settings are not synchronized (for example, the reserved management IP address).

 

4. Isolate the subordinate unit

Isolate of the subordinate unit from the network. From the subordinate unit GUI, go to System > Network > Interface, edit the traffic interfaces (in this example Internal2 and wan1) and set their Administrative Status to Down.

 

Isolate of the subordinate unit from the primary unit. Set the Administrative Status of the heartbeat interfaces (Internal4 and Internal5) to Down.

To avoid a split brain (when the heartbeat interfaces become disconnected and both cluster members become primary units) you must bring the traffic interfaces down before the heartbeat interfaces.

 

 

Check the System Information widget of the subordinate unit. It will think its the primary unit. Because its traffic interfaces are down, all traffic is going to the actual primary unit.

 

Connect to the primary unit GUI. The System Information widget should show just one cluster member.

 

5. Upgrade the cluster firmware and re-establish the cluster

Upgrade the firmware running on the primary unit (the one still processing traffic) using any normal firmware update procedure. For a short time during the upgrade network traffic is blocked. After the upgrade, make sure the primary unit is operating as expected. If it is not, go to step 6. Revert to the original firmware version.

Once you have done enough testing to establish that the primary unit is operating as expected with the new firmware, you can upgrade the subordinate unit to the same version. Log into the subordinate unit using its reserved management interface and upgrade the firmware.

Log into the primary unit reserved management interface and re-enable configuration synchronization.

config system ha
   set sync-config enable
end

Log into the subordinate unit, enable configuration synchronization, bring up its heartbeat interfaces and bring up its traffic interfaces.

The cluster resumes operating normally. You can use the get system ha status and diagnose sys ha status commands to verify that HA is operating normally.

Back up the configuration of the primary and subordinate FortiGate units. Backed up configuration files are specific to FortiOS versions.

6. Revert to the original firmware version

If the update didn’t succeed or the primary unit is not operating as expected, bring down the primary unit traffic interfaces and then heartbeat interfaces.

Then bring up the subordinate unit traffic and heartbeat interfaces. The subordinate unit, running the original firmware version, becomes the primary unit and processes traffic normally.

Downgrade the firmware running on the original primary unit to the original firmware version (that is currently running on the subordinate unit). Bring up the heartbeat and traffic interfaces on the original primary unit. The two cluster members re-establish the cluster, running the original firmware version.

For further reading, check out Configuring and connecting HA clusters in the FortiOS 5.2 Handbook.

Bill Dickie

Technical Writer at Fortinet
After completing a science degree at the University of Waterloo, Bill began his professional life teaching college chemistry in Corner Brook, Newfoundland and fell into technical writing after moving to Ottawa in the mid '80s. Tech writing stints at all sorts of companies finally led to joining Fortinet to write the first FortiGate-300 Administration Guide.
Facebooktwittergoogle_pluslinkedinFacebooktwittergoogle_pluslinkedin
For information about this configuration, check out High Availability with FGCP (Expert).