'NAT Hide failure - there are currently no available ports for hide operation' log appears repeatedly in SmartView Tracker

Product	Security Gateway, CoreXL, ClusterXL
Version	R70, R71, R75, R76, R77, R77.10, R77.20, R77.30

Symptoms

• 'NAT Hide failure - there are currently no available ports for hide operation' log for dropped Hide NATed connections appears repeatedly in SmartView Tracker.

Cause

Security Gateway, by default has 200 'high' ports and 20 'low' ports for Hide NAT (these numbers (200 and 20) are two parameters that are used by internal logic for static port allocation calculation). The whole range of ports is around 50 000, and Hide NAT is done per IP address. Hide NAT port exhaustion might be caused by the following on Security Gateway:

1. High number of CoreXL FW instances.When CoreXL is enabled on Security Gateway, in order to share the range of available ports between all CoreXL FW instances, this number of ports is divided by the number of CoreXL FW instances, thus each CoreXL FW instance has its own range. For example, on a Security Gateway with 10 CoreXL FW instances, each CoreXL FW instance will get 20 'high' ports and 2 'low' ports for Hide NAT.
2. 'HTTP/HTTPS Proxy' feature (introduced in R75.40 version).'HTTP/HTTPS Proxy' feature uses Hide NAT when opening the connection from the Security Gateway to the Destination.

 

Solution

(1) Background

In general, source ports for NAT functionality are divided into three ranges:

Range name	Port numbers	Comments
Low	600 - 1023	None
High	10000 - 60000	Used for standard connections (usually).
Extra, or Global	60001 - 65000	Used for features that are not supported by CoreXL (in which case, the traffic will be processed only by Core FW Instance #0).

Which port range will be used for the port allocation depends on the connection's service. Maximal number of concurrent hidden connections per destination is the combination of Destination "X" with NAT "Y" and the low/high port range according to the ranges division. Maximal number of concurrent connection using extra ports is the size of extra ports range. These port ranges are being allocated statically during policy installation. The port distribution is based on the following factors:

• Number of CoreXL FW instances
• Whether cluster is enabled and number of cluster members
• Whether SecureXL is enabled
• Whether VPN blade is enabled

Notes about SecureXL:

• When enabling SecureXL NAT Templates, SecureXL gets a share of port ranges from the ranges dedicated to the specific CoreXL FW instance.
• SecureXL range is static and cannot be shared.

The static allocation includes in its logic several verifications, which may fail the allocation and machine will face NAT problems. Two common issues:

• NAT is not being performed, and there is no log indicating NAT problems. In such case, change the value of 'hide_max_high_port'.
• NAT occasionally runs out of available ports, and a log is being sent with the description "NAT Hide failure - there are currently no available ports for hide operation". In such case - need to manipulate the static allocation by changing values of port quotas (as described below). Note that increasing these values too much may fail other verifications in the allocation and may also cause NAT problems.

Additional note: Sometimes NAT ports are being used even if customer does not use NAT - e.g., for cluster sync.  

(2) Procedure

The default limit of Hide NAT ports is controlled:

• By global kernel parameter 'fwx_high_port_quota' for 'high' ports (to check the current value, run fw ctl get int fwx_high_port_quota).
• By global kernel parameter 'fwx_low_port_quota' for 'low' ports (to check the current value, run fw ctl get int fwx_low_port_quota).

  The values of these kernel parameters are part of several internal parameters used for calculating the port ranges per CoreXL FW instance.

1. Set the desired value for 'fwx_high_port_quota'.
2. Set the desired value for 'fwx_low_port_quota'.
3. Reboot the Security Gateway.

   Example for Gaia / SecurePlatform OS:
   

   1. Create the $FWDIR/boot/modules/fwkern.conf file (if it does not exist): [Expert@HostName]# touch $FWDIR/boot/modules/fwkern.conf
   2. Edit the $FWDIR/boot/modules/fwkern.conf file in Vi editor: [Expert@HostName]# vi $FWDIR/boot/modules/fwkern.conf
   3. Add the following lines (spaces are not allowed): fwx_high_port_quota=VALUE fwx_low_port_quota=VALUE
   4. Save the changes in the file and exit from Vi editor.
   5. Reboot the Security Gateway.

 

(3) Notes

• The default values are:
  • 'fwx_high_port_quota' = 200
  • 'fwx_low_port_quota' = 20
  These values are part of several internal parameters used for calculating the port ranges per CoreXL FW instance.
• In cluster environment, these changes must be performed on all cluster members.