FortiGate-VM HA for RedHat OpenStack 10 – Expert

Facebooktwittergoogle_plusredditpinterestlinkedinFacebooktwittergoogle_plusredditpinterestlinkedin

This example shows how to set up FortiGate Clustering Protocol (FGCP) HA with two FortiGate VMs in a RedHat OpenStack 10 environment. The example includes the two FortiGate VMs connected to a private network (private01). The FortiGate-VMs protect two networks (network-l and network-r). Each network includes a CirrOS instance (cirros-l and cirros-r) for testing.

To support HA heartbeat communication, the OpenStack environment also includes a network named ha-sync configured with the subnet used by the HA heartbeat interfaces (169.254.0.0/24).

1. Setting up the networks in OpenStack

From the OpenStack environment command line, enter the following commands to create network-r and network-l and the ha-sync network.

$ source overcloudrc_tenant01

$ openstack network create network-r

$ openstack subnet create subnet-r --network network-r --subnet-range 172.32.0.0/24 --dns-nameserver 208.91.112.53

$ openstack network create network-l

$ openstack subnet create subnet-l --network network-l --subnet-range 172.33.0.0/24 --dns-nameserver 208.91.112.53

$ openstack network create ha-sync

$ openstack subnet create subnet-ha --network ha-sync --subnet-range 169.254.0.0/24 --dns-nameserver 208.91.112.53
Add the CirrOS instances to network-r and network-l:
$ openstack server create --flavor m1.tiny --image cirros035 --security-group web --nic net-id=network-r  cirros-r

$ openstack server create --flavor m1.tiny --image cirros035 --security-group web --nic net-id=network-l  cirros-l

2. Deploy two FortiGate-VMs

From the OpenStack command line, enter the following commands to deploy two FortiGate-VM instances (fgt-vm-1 and fgt-vm-2). These commands use the standard license files you receive when you register your FortiGate-VMs (in this example, FGVM080000103268.lic and FGVM080000109643.lic).

$ openstack server create --flavor m1.fortigate --image fgtb1486 --user-data /home/stack/openstack/cloud-init/userdata.txt --config-drive=true --file license=/home/stack/FG-licenses/FGVM080000103268.lic --security-group web --nic net-id=private01 --nic net-id=network-r --nic net-id=network-l --nic net-id=ha-sync fgt-vm-1

$ openstack server create --flavor m1.fortigate --image fgtb1486 --user-data /home/stack/openstack/cloud-init/userdata.txt --config-drive=true --file license=/home/stack/FG-licenses/FGVM080000109643.lic --security-group web --nic net-id=private01 --nic net-id=network-r --nic net-id=network-l --nic net-id=ha-sync fgt-vm-2

Here is an example userdata.txt file used for fgt-vm-1. The userdata.txt file for fgt-vm-2 would be the same except for the hostname.

The userdata.txt file allows you to set up a FortiGate-VM with a basic default configuration customized for your environment and requirements. This example configures interfaces, adds a DNS server, and adds two firewall policies that allow any traffic to pass between the port2 and port3 interfaces. These policies make it easier to test HA failover.

In addition, the MTU of the port4 interface is set to be compatible with the OpenStack 10 environment. By default, OpenStack 10 networks have an MTU of 1446. The userdata.txt file sets the MTU of port4 to 1400. This setting is required for the HA heartbeat interfaces to be able to communicate effectively over the ha-sync network. 

#FGT VM Config File

config sys global
set hostname fgt-vm-1
end
config system interface
edit port1
set mode dhcp
set allowaccess http https ssh ping
next
edit port2
set mode dhcp
set defaultgw disable
set allowaccess http https ssh ping
next
edit port3
set mode dhcp
set defaultgw disable
set allowaccess http https ssh ping
next
edit port4
set mtu-override enable
set mtu 1400
next
end
config system dns
set primary 208.91.112.53
end
config firewall policy
edit 1
set name "Allow port2 to port3"
set dstintf "port2"
set srcintf "port3"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
set nat enable
next
edit 2
set name "Allow port3 to port2"
set dstintf "port3" 
set srcintf "port2" 
set srcaddr "all" 
set dstaddr "all" 
set action accept 
set schedule "always" 
set service "ALL" 
set nat enable
end
config system central-management
set include-default-servers disable
set type fortimanager
set fmg 10.210.8.25
config server-list
edit 1
set server-type update rating
set server-address 10.210.8.25
end
end

You can use the OpenStack Horizon Networks view to verify the MTU assigned to the ha-sync network.

 

3. Disable port security for the FortiGate-VM and CirrOS instances

Use the RedHat OpenStack Horizon Instances view to verify the IP addresses of the FortiGate-VM, the CirrOS instances, and the networks the interfaces are connected to. For example:

 

From the OpenStack command line, run the following bash script to disable port security on the FG-VM interfaces.

#!/bin/bash
echo
echo 'Disable port_security on fgt-vm-1'
echo
echo
`source /home/stack/overcloudrc_tenant01`
FGT='fgt-vm-1'
IPADDR=`openstack server show $FGT | grep addresses | awk -F "|" '{print $3}' | awk -F "=" '{print $2}' | awk -F ";" '{print $1}'`
PORTID=`openstack port list | grep $IPADDR | awk -F "|" '{print $2}'`
neutron port-update $PORTID --no-security-groups --port_security_enabled=False
echo
echo $IPADDR
echo `openstack port show $PORTID`
echo
IPADDR=`openstack server show $FGT | grep addresses | awk -F "|" '{print $3}' | awk -F "=" '{print $3}' | awk -F ";" '{print $1}'`
PORTID=`openstack port list | grep $IPADDR | awk -F "|" '{print $2}'`
`neutron port-update $PORTID --no-security-groups --port_security_enabled=False`
echo
echo $IPADDR
echo `openstack port show $PORTID`
echo
IPADDR=`openstack server show $FGT | grep addresses | awk -F "|" '{print $3}' | awk -F "=" '{print $4}' | awk -F ";" '{print $1}'`
PORTID=`openstack port list | grep $IPADDR | awk -F "|" '{print $2}'`
`neutron port-update $PORTID --no-security-groups --port_security_enabled=False`
echo
echo $IPADDR
echo `openstack port show $PORTID`
echo
IPADDR=`openstack server show $FGT | grep addresses | awk -F "|" '{print $3}' | awk -F "=" '{print $5}' | awk -F ";" '{print $1}'`
PORTID=`openstack port list | grep $IPADDR | awk -F "|" '{print $2}'`
`neutron port-update $PORTID --no-security-groups --port_security_enabled=False`
echo
echo $IPADDR
echo `openstack port show $PORTID`
echo
echo 'Disable port-security on fgt-vm-2'
echo
FGT='fgt-vm-2'
IPADDR=`openstack server show $FGT | grep addresses | awk -F "|" '{print $3}' | awk -F "=" '{print $2}' | awk -F ";" '{print $1}'`
PORTID=`openstack port list | grep $IPADDR | awk -F "|" '{print $2}'`
neutron port-update $PORTID --no-security-groups --port_security_enabled=False
echo
echo $IPADDR
echo `openstack port show $PORTID`
echo
IPADDR=`openstack server show $FGT | grep addresses | awk -F "|" '{print $3}' | awk -F "=" '{print $3}' | awk -F ";" '{print $1}'`
PORTID=`openstack port list | grep $IPADDR | awk -F "|" '{print $2}'`
`neutron port-update $PORTID --no-security-groups --port_security_enabled=False`
echo
echo $IPADDR
echo `openstack port show $PORTID`
echo

IPADDR=`openstack server show $FGT | grep addresses | awk -F "|" '{print $3}' | awk -F "=" '{print $4}' | awk -F ";" '{print $1}'`
PORTID=`openstack port list | grep $IPADDR | awk -F "|" '{print $2}'`
`neutron port-update $PORTID --no-security-groups --port_security_enabled=False`
echo
echo $IPADDR
echo `openstack port show $PORTID`
echo
IPADDR=`openstack server show $FGT | grep addresses | awk -F "|" '{print $3}' | awk -F "=" '{print $5}' | awk -F ";" '{print $1}'`
PORTID=`openstack port list | grep $IPADDR | awk -F "|" '{print $2}'`
`neutron port-update $PORTID --no-security-groups --port_security_enabled=False`
echo
echo $IPADDR
echo `openstack port show $PORTID`
echo

From the OpenStack command line, associate floating IPs to the two FortiGate-VMs, by entering the following commands:

openstack server add floating ip fgt-vm-1 10.210.9.10 

openstack server add floating ip fgt-vm-2 10.210.9.14

4. Complete the FortiGate-VM configuration

From each FortiGate-VM instance CLI, enter the following commands to change the FortiGate-VM interfaces from DHCP to static, add IP addresses, and add a static route. The IP addresses assigned to the interfaces must be on the subnets of the networks the interfaces are connected to.

The example shows the fgt-vm-1 configuration. The fgt-vm-2 configuration would be the same except for the interface IP addresses.

config system interface
   edit "port1"
    set mode static
       set ip 172.31.0.3 255.255.255.0
       set allowaccess ping https ssh http
   next
   edit "port2"
    set mode static
       set ip 172.32.0.9 255.255.255.0
       set allowaccess ping https ssh http
   next
   edit "port3"
    set mode static
       set ip 172.33.0.4 255.255.255.0
       set allowaccess ping https ssh http
   next
end

config router static
   edit 1
       set gateway 172.31.0.1
       set device "port1"
   next
end

From each FortiGate-VM instance CLI, configure both FortiGate-VMs for HA. Both FortiGate-VMs must have the same HA configuation.

config system ha    
   set group-name "group-01"    
   set mode a-p    
   set password <password>    
   set hbdev "port4" 50    
   set override disable    
   set monitor "port2" 
end

5. CirrOS instance configuration

From each CirrOS CLI, configure each CirrOS instance with a default gateway that points at the FortiGate-VM interface connected to the same network as the CirrOS instance. Enter the following commands from each CirrOS CLI:

sudo route del default 
sudo ip route add default via <FG-IP-Address>

6. Testing cluster operation and failover

On the cirros-l instance console, start a continuous ping to the IP address of cirros-r. On the cirros-r instance console, start a continuous ping to the IP address of cirros-l:

On both FortiGate-VMs, use the command diagnose sniffer packet any 'icmp' 4 to sniff ICMP packets. You should only see packets going through the primary unit.

Now shut down the primary unit. You can do this from the OpenStack Horizon Instances list.

After failover, on the new primary unit enter the command diagnose sniffer packet any 'icmp' 4 to verify that the pings are now going through it.

7. Troubleshooting diagnose commands

On either FortiGate-VM you can use the diagnose sys ha status command to verify the status of the cluster.

 
fgt-vm # diagnose sys ha status
HA information
Statistics
    traffic.local = s:0 p:42311 b:9008646
    traffic.total = s:0 p:42316 b:9009528
    activity.fdb  = c:0 q:0

Model=80008, Mode=2 Group=0 Debug=0
nvcluster=1, ses_pickup=0, delay=0

[Debug_Zone HA information]
HA group member information: is_manage_master=1.
FGVM080000109643: Master, serialno_prio=0, usr_priority=128, hostname=fgt-vm
FGVM080000103268:  Slave, serialno_prio=1, usr_priority=128, hostname=fgt-vm

[Kernel HA information]
vcluster 1, state=work, master_ip=169.254.0.1, master_id=0:
FGVM080000109643: Master, ha_prio/o_ha_prio=0/0
FGVM080000103268:  Slave, ha_prio/o_ha_prio=1/1
The command get system ha status shows similar information.
fgt-vm # get system ha status
HA Health Status: OK
Model: FortiGate-VM64-KVM
Mode: HA A-P
Group: 0
Debug: 0
Cluster Uptime: 0 days 02:04:26
Cluster state change time: 2017-09-01 03:08:19
Master selected using:
   <2017/09/01 03:08:19> FGVM080000109643 is selected as the master because it has the largest value of serialno.
ses_pickup: disable
override: disable
Configuration Status:
   FGVM080000109643(updated 2 seconds ago): in-sync
   FGVM080000103268(updated 0 seconds ago): out-of-sync
System Usage stats:
   FGVM080000109643(updated 2 seconds ago):
       sessions=4, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=55%
   FGVM080000103268(updated 0 seconds ago):
       sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=54%
HBDEV stats:
   FGVM080000109643(updated 2 seconds ago):
       port4: physical/10000full, up, rx-bytes/packets/dropped/errors=15043566/61878/0/0, tx=158364378/146977/0/0
   FGVM080000103268(updated 0 seconds ago):
       port4: physical/10000full, up, rx-bytes/packets/dropped/errors=29442835/61625/49/0, tx=25246662/68626/0/0
MONDEV stats:
   FGVM080000109643(updated 2 seconds ago):
       port2: physical/10000full, up, rx-bytes/packets/dropped/errors=1892/8/0/0, tx=173710/307/0/0
   FGVM080000103268(updated 0 seconds ago):
       port2: physical/10000full, up, rx-bytes/packets/dropped/errors=174390/306/0/0, tx=2352/13/0/0
Master: fgt-vm          , FGVM080000109643
Slave : fgt-vm          , FGVM080000103268
number of vcluster: 1
vcluster 1: work 169.254.0.1
Master:0 FGVM080000109643
Slave :1 FGVM080000103268
The command diagnose system ha checksum show shows whether the configurations of the FortiGate-VMs in the cluster are synchronized. If the configurations are synchronized both sets of checksums should match.
fgt-vm # diagnose sys ha checksum show
is_manage_master()=1, is_root_master()=1
debugzone
global: 33 6f ee 5b 78 a5 22 84 39 ec 36 d3 1c 54 7c 78
root: 40 0d fb 04 12 41 df ad f1 64 14 03 ff ec f5 01
all: d3 2f 6f bb a6 e7 77 db 27 75 81 b2 94 f3 fd 68

checksum
global: 33 6f ee 5b 78 a5 22 84 39 ec 36 d3 1c 54 7c 78
root: 40 0d fb 04 12 41 df ad f1 64 14 03 ff ec f5 01
all: d3 2f 6f bb a6 e7 77 db 27 75 81 b2 94 f3 fd 68
If the checksums do not match, you can use the diagnose sys ha checksum show and diagnose sys ha checksum show global  commands to show more detailed checksum results, the following example shows the first few lines of output of the diagnose sys ha checksum show global command:
diagnose sys ha checksum show global
system.global: 2c79958c132639dfe61ab782a2f213ec
system.accprofile: 7d79452c78377be2616149264a18fd5c
system.vdom-link: 00000000000000000000000000000000
wireless-controller.inter-controller: 00000000000000000000000000000000
wireless-controller.global: 00000000000000000000000000000000
wireless-controller.vap: 00000000000000000000000000000000
system.switch-interface: 00000000000000000000000000000000
system.interface: 8690699bc33c7c15b20e017876cf1e37
...
If the configurations are synchronized all of the checksums displayed using these commands from both of the FortiGate-VMs should match. If they do not you can use the output to see what parts of the configuration are not synchronized.

 

Bill Dickie

Our Fearless Documentation Leader at Fortinet
After completing a science degree at the University of Waterloo, Bill began his professional life teaching college chemistry in Corner Brook, Newfoundland and fell into technical writing after moving to Ottawa in the mid '80s. Tech writing stints at all sorts of companies finally led to joining Fortinet to write the first FortiGate-300 Administration Guide.
  • Was this helpful?
  • Yes   No
In RedHat OpenStack, the networking part of OpenStack (called Neutron) only allows traffic with known IP/MAC address combinations. This makes the network very secure. With a firewall, it isn’t possible to provide all the possible IP/MAC address combinations that might pass through the network. To allow normal traffic this step describes how to disable port security for the FortiGate-VMs. For more information, see Managing port level security in OpenStack.