Management HA changes to Lagging state shortly after installing policy

Product	Security Management, Multi-Domain Management / Provider-1
Version	R75.40, R76, R77, R77.10

Symptoms

• Shortly after policy installation Management HA changes it's status to Lagging
• Smartview Tracker shows an Audit log that contracts were created and deleted by Administrator - SmartUpdate.
• This log comes from the Primary Smartcenter server only.

Cause

After policy install IPS contracts are checked when IPS is enabled and automatic updates are configured. This is normal behavior.
During this check the contracts are deleted and recreated again, as shown by the Audit logs.
This change is only made on the Primary Smartcenter server. It also does not trigger a synchronization event. Therefore the Secondary Smartcenter server is Lagging after this update.

Solution

This problem was fixed. The fix is included in:

• Check Point R77.20 

_____
 

If you choose not to upgrade to R77.20, you can use this workaround:

The following procedure resolves the contract checking issue for Management HA. This is the recommended procedure and is scheduled to be included in future releases.

==============
On both HA members:

1. Stop serivces
# cpstop
2. locate tables.C
# cd $FWDIR/conf
3. backup tables.C
# cp tables.C tables.C.orig
4. add attribute mgmt_ha_opt to contracts table.
# vi tables.C
(Search for ": (contracts" section, and add the following line just after ":private_lockable (false)")
-----
:mgmt_ha_opt (0x00000001)
-----

Following is contracts table example (after adding mgmt_ha_opt line):
----------
: (contracts
:display_str (Contracts)
:table_clsid ("{4D998D0A-25DC-48f6-9677-8FAB0F991BF2}")
:db_clsid ("{C7E765A8-CBD2-4b04-9F7F-7EC61F4ECBB5}")
:file_name (contracts.c)
:archive_opt (0xfffffff9)
:read_permission (0x00000000)
:write_permission (0x00040000)
:read_permissions_list ("{all}")
:write_permissions_list ("{objects_database}")
:private_lockable (false)
:mgmt_ha_opt (0x00000001)
:queries (
:all ("*")
)
)
----------
5. Save and exit file
6. Start service
# cpstart

'NAT Hide failure - there are currently no available ports for hide operation' log appears repeatedly in SmartView Tracker

Product	Security Gateway, CoreXL, ClusterXL
Version	R70, R71, R75, R76, R77, R77.10, R77.20, R77.30

Symptoms

• 'NAT Hide failure - there are currently no available ports for hide operation' log for dropped Hide NATed connections appears repeatedly in SmartView Tracker.

Cause

Security Gateway, by default has 200 'high' ports and 20 'low' ports for Hide NAT (these numbers (200 and 20) are two parameters that are used by internal logic for static port allocation calculation). The whole range of ports is around 50 000, and Hide NAT is done per IP address. Hide NAT port exhaustion might be caused by the following on Security Gateway:

1. High number of CoreXL FW instances.When CoreXL is enabled on Security Gateway, in order to share the range of available ports between all CoreXL FW instances, this number of ports is divided by the number of CoreXL FW instances, thus each CoreXL FW instance has its own range. For example, on a Security Gateway with 10 CoreXL FW instances, each CoreXL FW instance will get 20 'high' ports and 2 'low' ports for Hide NAT.
2. 'HTTP/HTTPS Proxy' feature (introduced in R75.40 version).'HTTP/HTTPS Proxy' feature uses Hide NAT when opening the connection from the Security Gateway to the Destination.

 

Solution

(1) Background

In general, source ports for NAT functionality are divided into three ranges:

Range name	Port numbers	Comments
Low	600 - 1023	None
High	10000 - 60000	Used for standard connections (usually).
Extra, or Global	60001 - 65000	Used for features that are not supported by CoreXL (in which case, the traffic will be processed only by Core FW Instance #0).

Which port range will be used for the port allocation depends on the connection's service. Maximal number of concurrent hidden connections per destination is the combination of Destination "X" with NAT "Y" and the low/high port range according to the ranges division. Maximal number of concurrent connection using extra ports is the size of extra ports range. These port ranges are being allocated statically during policy installation. The port distribution is based on the following factors:

• Number of CoreXL FW instances
• Whether cluster is enabled and number of cluster members
• Whether SecureXL is enabled
• Whether VPN blade is enabled

Notes about SecureXL:

• When enabling SecureXL NAT Templates, SecureXL gets a share of port ranges from the ranges dedicated to the specific CoreXL FW instance.
• SecureXL range is static and cannot be shared.

The static allocation includes in its logic several verifications, which may fail the allocation and machine will face NAT problems. Two common issues:

• NAT is not being performed, and there is no log indicating NAT problems. In such case, change the value of 'hide_max_high_port'.
• NAT occasionally runs out of available ports, and a log is being sent with the description "NAT Hide failure - there are currently no available ports for hide operation". In such case - need to manipulate the static allocation by changing values of port quotas (as described below). Note that increasing these values too much may fail other verifications in the allocation and may also cause NAT problems.

Additional note: Sometimes NAT ports are being used even if customer does not use NAT - e.g., for cluster sync.  

(2) Procedure

The default limit of Hide NAT ports is controlled:

• By global kernel parameter 'fwx_high_port_quota' for 'high' ports (to check the current value, run fw ctl get int fwx_high_port_quota).
• By global kernel parameter 'fwx_low_port_quota' for 'low' ports (to check the current value, run fw ctl get int fwx_low_port_quota).

  The values of these kernel parameters are part of several internal parameters used for calculating the port ranges per CoreXL FW instance.

1. Set the desired value for 'fwx_high_port_quota'.
2. Set the desired value for 'fwx_low_port_quota'.
3. Reboot the Security Gateway.

   Example for Gaia / SecurePlatform OS:
   

   1. Create the $FWDIR/boot/modules/fwkern.conf file (if it does not exist): [Expert@HostName]# touch $FWDIR/boot/modules/fwkern.conf
   2. Edit the $FWDIR/boot/modules/fwkern.conf file in Vi editor: [Expert@HostName]# vi $FWDIR/boot/modules/fwkern.conf
   3. Add the following lines (spaces are not allowed): fwx_high_port_quota=VALUE fwx_low_port_quota=VALUE
   4. Save the changes in the file and exit from Vi editor.
   5. Reboot the Security Gateway.

 

(3) Notes

• The default values are:
  • 'fwx_high_port_quota' = 200
  • 'fwx_low_port_quota' = 20
  These values are part of several internal parameters used for calculating the port ranges per CoreXL FW instance.
• In cluster environment, these changes must be performed on all cluster members.

 

How do I debug VPND on Check Point ?

1. Check if connectivity exist between the 2 Gateway peers
2. VPN Debugging - Looking at the IKE negoatations
3. Can both sides see the IKE packets arriving during teh Key Exchange?

• IKE Process (2 Phases) 
• Phase 1 - Main Mode (6 Packets)
• Phase 2 - Quick Mode (3 Packets)
• Turn VPN Debug On - enter the command "vpn debug on; vpn debug ikeon" or "vpn debug trunc".
• The $FWDIR/log/ike.elg file contains information once debugging is enabled. Checkpoint has a tool IKEView.exe - it parse information of ike.elg
  
• If you run # vpn debug mon, the output file is ikemonitor.snoop. In this output file, all the IKE payloads are in clear. Whereas, in monitor.out, all the IKE payloads are encrypted. 
  Phase I
• Negotiates encryption methods (DES/3DES/AES etc)
• The key length
• The Hash Algorithm (MD5/SHA1)
• Creates a key to protect the messages of the exchange.

It does this in 5 stages:

1. Peers Authenticate using Certificates or a pre-shared secret.
2. Each peer generates a private Diffie-Hellman key from random bits and from that derives a DH public key. These are then exchanged.
3. Each peer generates a shared secret from its private key and its peers public key, this is the DH key.
4. The peers exchange DH Key material (random bits and mathematical data) and methods for PhaseII are agreed for encryption and integrity.

Each side generates a symmetric key (based upon the DH key and key material exchanged).

In IkeView under the IP address of the peer, open the Main Mode Packet 1 - expand :
> "P1 Main Mode ==>" for outgoing or "P1 Main Mode <==" for incoming > MM Packet 1 > Security Association > prop1 PROTO_ISAKMP > tran1 KEY_IKE

UNDERSTAND THE 5 PACKETS

1. If your encryption fails in Main Mode Packet 1, then you need to check your VPN communities.
2. Packet 2 ( MM Packet 2 in the trace ) is from the responder to agree on one encryption and hash algorithm.
3. Packets 3 and 4 arent usually used when troublshooting. They perform key exchanges and include a large number called a NONCE. The NONCE is a set of never before used random numbers sent to the other part, signed and returned to prove the parties identity.
4. Packets 5 and 6 perform the authentication between the peers. The peers IP address shows in the ID field under MM packet 5. Packet 6 shows that the peer has agreed to the proposal and has authorised the host initiating the key exchange.

NOTE:
1. If your encryption fails in Main Mode Packet 1, then you need to check your VPN communities.
2. If your encryption fails in Main Mode Packet 5, then you need to check the authentication - Certificates or pre-shared secrets

PHASE II
The IPSec Security Associations (SAs) are negotiated,

- The shared secret key material used for the SA is determined and there is an additional DH exchange.
- Phase II failures are generatlly due to a misconfigured VPN domain.
- Phase II occurs in 3 stages:
1. Peers exchange key material and agree encryption and integrity methods for IPSec.
2. The DH key is combined with the key material to produce the symmetrical IPSec key.
3. Symmetric IPSec keys are generated.   To debug VPND run the following command :

vpn debug trunc

To disable the debug run the commands :

vpn debug off; vpn debug ikeoff

To view the logs run the command :

cd $FWDIR/log ; tail -f ike.elg vpnd.elg

show commands

save config	save the current configuration
show commands	shows all commands
show allowed-client all	show allowed clients
show arp dynamic all	displays the dynamic arp entries
show arp proxy all	shows proxy arp
show arp static all	displays all the static arp entry
show as	displays autonomous system number
show assets all	display hardware information
show bgp stats	shows bgp statistics
show bgp summary	shows summary information about bgp
show vrrp stats	show vrrp statistics
show bootp stats	shows bootp/dhcp relay statistics
show bootp interface	show all bootp/dhcp relay interfaces
show bonding group	show all bonding groups
show bridging groups	show all bridging groups
show backups	shows a list of local backups
show backup status	show the status of a backup or restore operation being performed
show backup last-successful	show the latest successful backup
show backup logs	show the logs of the recent backups/restores performed
show clock	show current clock
show configuration	show configuration
show-config state	shows the state of configuration either saved or unsaved
show date	shows date
show dns primary	shows primary dns server
show dns secondary	shows secondary dns server
show extended commands	shows all extended commands
show groups	shows all user groups
show hostname	show host name
show inactivity-timeout	shows inactivity-timeout settings
show interfaces	shows all interfaces
show interfaces ethx	shows settings related to an interface “x
show interfaces	show detailed information about all interfaces
show ipv6-state	shows ipv6 status as enabled or disabled
show management interface	shows management interface configuration
show ntp active	shows ntp status as enabled or disabled
show ntp servers	shows ntp servers
show ospf database	shows ospf database information
show ospf neighbors	shows ospf neighbors information
show ospf summary	shows ospf summary information
show pbr rules	shows policy based routing rules
show pbr summary	shows policy based routing summary information
show pbr tables	show pbr tables
show route	shows routing table
show routed version	shows information about routed version
show snapshots	shows a list of local snapshots
show snmp agent-version	shows whether the version is v1/v2/v3
show snmp interfaces	shows snmp agent interface
show snmp traps receivers	shows snmp trap receivers
show time	shows local machine time
show timezone	show configured timezone
show uptime	show system uptime
show users	show configured users and their homedir, uid/gid and shell
show user <username>	shows settings related to a particular user
show version all	shows version related to os edition, kernel version, product version etc
show virtual-system all	show virtual-systems configured
show vpn tunnels	use to show the vpn tunnels
show vrrp stats	shows vrrp status
show vrrp interfaces	shows vrrp enabled interfaces

set commands

add allowed-client host any-host / add allowed-client host <ip address>	add any host to the allowed clients list/ add allowed client by ipv4 address
add backup local	create and store a backup file in /var/cpbackups/backups/( on open servers) or /var/log/cpbackup/backups/ ( on checkpoint appliances)
add backup scp ip value path value username value	adds backup to scp server
add backup tftp ip value [ interactive ]	adds backup to tftp server
add snapshot	create snapshots which backs up everything like os configuration, checkpoint configuration, versions, patch level), including the drivers
add syslog log-remote-address <ip address> level <emerg/alert/crit/err/warning/notice/info/debug/all>	specifies syslog parameters
add user <username> uid <user-id-value> homedir	creates a user
expert	executes system shell
halt	put system to halt
history	shows command history
lock database override	overrides the config-lock settings
quit	exits out of a shell
reboot	reboots a system
restore backup local [value]	restores local backup interactively
rollback	ends the transaction mode by reverting the changes made during transaction
save config	save the current configuration
set backup restore local <filename>	restores a local backup
set core-dump <enable/disable>	enable/disable core dumps
set date yyyy-mm-dd	sets system date
set dhcp server enable	enable dhcp server
set dns primary <x.x.x.x>	sets primary dns ip address
set dns secondary <x.x.x.x>	sets secondary dns ip address
set expert-password	set or change password for entering into expert mode
set edition default <value>	set the default edition to 32-bit or 64-bit
set hostname <value>	sets system hostname
set inactivity-timeout <value>	sets the inactivity timeout
set interface ethx ipv4-address x.x.x.x mask-length 24	adds ip address to an interface
set ipv6-state on/off	sets ipv6 status as on or off
set kernel-routes on/off	sets kernel routes to on/off state
set management interface <interface name>	sets an interface as management interface
set message motd value	sets message of the day
set ntp active on/off	activates ntp on/off
set ntp server primary x.x.x.x version <1/2/3/4>	sets primary ntp server
set ntp server secondary x.x.x.x version <1/2/3/4>	sets secondary ntp server
set snapshot revert<filename>	revert the machine to the selected snapshot
set snmp agent on/off	sets the snmp agent daemon on/off
set snmp agent-version <value>	sets snmp agent version
set snmp community <value> read-only	sets snmp readonly community string
add snmp interface <interface name>	sets snmp agent interface
set snmp traps receiver <ip address> version v1 community value	specifies trap receiver
set snmp traps trap <value>	set snmp traps
set static-route x.x.x.x/24 nexthop gateway address x.x.x.x on	adds specific static route
set time <value>	sets system time
set time zone <time-zone>	sets the time zone
set vsx off	sets vsx mode on
set vsx on	sets vsx mode off
set user <username> password	sets users password
set web session-timeout <value>	sets web configuration session time-out in minutes
set web ssl-port <value>	sets the web ssl-port for the system

Commands missing, want to add a command to the list ? Just add them to the comments section and we will include them.

Check Point Commands

CP, FW & FWM

cphaprob stat	List cluster status
cphaprob -a if	List status of interfaces
cphaprob syncstat	shows the sync status
cphaprob list	Shows a status in list form
cphastart/stop	Stops clustering on the specfic node
cp_conf sic	SIC stuff
cpconfig	config util
cplic print	prints the license
cprestart	Restarts all Check Point Services
cpstart	Starts all Check Point Services
cpstop	Stops all Check Point Services
cpstop -fwflag -proc	Stops all checkpoint Services but keeps policy active in kernel
cpwd_admin list	List checkpoint processes
cplic print	Print all the licensing information.
cpstat -f all polsrv	Show VPN Policy Server Stats
cpstat	Shows the status of the firewall
fw tab -t sam_blocked_ips	Block IPS via SmartTracker
fw tab -t connections -s	Show connection stats
fw tab -t connections -f	Show connections with IP instead of HEX
fw tab -t fwx_alloc -f	Show fwx_alloc with IP instead of HEX
fw tab -t peers_count -s	Shows VPN stats
fw tab -t userc_users -s	Shows VPN stats
fw checklic	Check license details
fw ctl get int [global kernel parameter]	Shows the current value of a global kernel parameter
fw ctl set int [global kernel parameter] [value]	Sets the current value of a global keneral parameter. Only Temp ; Cleared after reboot.
fw ctl arp	Shows arp table
fw ctl install	Install hosts internal interfaces
fw ctl ip_forwarding	Control IP forwarding
fw ctl pstat	System Resource stats
fw ctl uninstall	Uninstall hosts internal interfaces
fw exportlog .o	Export current log file to ascii file
fw fetch	Fetch security policy and install
fw fetch localhost	Installs (on gateway) the last installed policy.
fw hastat	Shows Cluster statistics
fw lichosts	Display protected hosts
fw log -f	Tail the current log file
fw log -s -e	Retrieve logs between times
fw logswitch	Rotate current log file
fw lslogs	Display remote machine log-file list
fw monitor	Packet sniffer
fw printlic -p	Print current Firewall modules
fw printlic	Print current license details
fw putkey	Install authenication key onto host
fw stat -l	Long stat list, shows which policies are installed
fw stat -s	Short stat list, shows which policies are installed
fw unloadlocal	Unload policy
fw ver -k	Returns version, patch info and Kernal info
fwstart	Starts the firewall
fwstop	Stop the firewall
fwm lock_admin -v	View locked admin accounts
fwm dbexport -f user.txt	used to export users , can also use dbimport
fwm_start	starts the management processes
fwm -p	Print a list of Admin users
fwm -a	Adds an Admin
fwm -r	Delete an administrator

Provider 1

mdsenv [cma name]	Sets the mds environment
mcd	Changes your directory to that of the environment.
mds_setup	To setup MDS Servers
mdsconfig	Alternative to cpconfig for MDS servers
mdsstat	To see the processes status
mdsstart_customer [cma name]	To start cma
mdsstop_customer [cma name]	To stop cma
cma_migrate	To migrate an Smart center server to CMA
cmamigrate_assist	If you dont want to go through the pain of tar/zip/ftp and if you wish to enable FTP on Smart center server

VPN

vpn tu	VPN utility, allows you to rekey vpn
vpn ipafile_check ipassignment.conf detail‏	Verifies the ipassignment.conf file
dtps lic	show desktop policy license status
cpstat -f all polsrv	show status of the dtps
vpn shell /tunnels/delete/IKE/peer/[peer ip]	delete IKE SA
vpn shell /tunnels/delete/IPsec/peer/[peer ip]	delete Phase 2 SA
vpn shell /show/tunnels/ike/peer/[peer ip]	show IKE SA
vpn shell /show/tunnels/ipsec/peer/[peer ip]	show Phase 2 SA
vpn shell show interface detailed [VTI name]	show VTI detail

Debugging

fw ctl zdebug drop

shows dropped packets in realtime / gives reason for drop

SPLAT Only

router	Enters router mode for use on Secure Platform Pro for advanced routing options
patch add cd	Allows you to mount an iso and upgrade your checkpoint software (SPLAT Only)
backup	Allows you to preform a system operating system backup
restore	Allows you to restore your backup
snapshot	Performs a system backup which includes all Check Point binaries. Note : This issues a cpstop.

VSX

vsx get [vsys name/id]	get the current context
vsx set [vsys name/id]	set your context
fw -vs [vsys id] getifs	show the interfaces for a virtual device
fw vsx stat -l	shows a list of the virtual devices and installed policies
fw vsx stat -v	shows a list of the virtual devices and installed policies (verbose)
reset_gw	resets the gateway, clearing all previous virtual devices and settings.

Understanding the output of "fw monitor"

Suppose there is a capture to see all packets going to or coming from 9.9.9.9. The capture statement would look like this:

fw monitor -e 'accept (([12:4,b]=9.9.9.9) or ([16:4,b]=9.9.9.9));'

Then we have an inside host attempt to go to that IP over port 80. We will see the following output:

[vs_0][fw_9] eth0:i[52]: 10.0.0.1 -> 9.9.9.9 (TCP) len=52 id=6901
TCP: 6437 -> 80 .S.... seq=3d4cd035 ack=00000000
[vs_0][fw_9] eth0:I[52]: 10.0.0.1 -> 9.9.9.9 (TCP) len=52 id=6901
TCP: 6437 -> 80 .S.... seq=3d4cd035 ack=00000000
[vs_0][fw_9] eth1:o[52]: 10.0.0.1 -> 9.9.9.9 (TCP) len=52 id=6901
TCP: 6437 -> 80 .S.... seq=3d4cd035 ack=00000000
[vs_0][fw_9] eth1:O[52]: 11.11.11.11 -> 9.9.9.9 (TCP) len=52 id=6901
TCP: 27030-> 80 .S.... seq=3d4cd035 ack=00000000

• eth0 means the packet is being processed on the Lan1 interface
• eth0:i indicates this packet is captured pre-inbound rules
• eth0:I indicates this packet is captured post-Inbound rules
• eth1 indicates the interface the packet will be routed out of
• eth1:o indicates the packet was captured pre-outbound rules
• eth1:O indicates the packet was captured post-Outbound rules.
• UDP: 27030-> 80indicates this packet is a UDP packet with a source port of 27030 and destination port of 80. You can see the source IP has changed due to NAT during the outbound rules.
• .S.... indicates this packet has the SYN flag set

Now let’s look at the return packet, the SYN-ACK packet and how it looks coming back through the firewall:

[vs_0][fw_9] eth1:i[52]: 9.9.9.9 -> 11.11.11.11 (TCP) len=52 id=0
TCP: 80 -> 46127 .S..A. seq=8348a727 ack=3d4cd036
[vs_0][fw_9] eth1:I[52]: 9.9.9.9 -> 10.0.0.1 (TCP) len=52 id=0
TCP: 80 -> 6437 .S..A. seq=8348a727 ack=3d4cd036
[vs_0][fw_9] eth0:o[52]: 9.9.9.9 -> 10.0.0.1 (TCP) len=52 id=0
TCP: 80 -> 6437 .S..A. seq=8348a727 ack=3d4cd036
[vs_0][fw_9] eth0:O[52]: 9.9.9.9 -> 10.0.0.1 (TCP) len=52 id=0
TCP: 80 -> 6437 .S..A. seq=8348a727 ack=3d4cd036

• .S..A. indicates this packet has the SYN ACK flags set.

SmartEvent shows no new events

Product	SmartEvent / Eventia Analyzer
Version	R75.45, R75.46, R75.47, R77.10, R77.20, R77.30

 Symptoms

• SmartEvent shows no new events, even though logs are being produced and analyzed by the Correlation Unit.
• postgres process appears as 'idle'.
• cpsemd.elg shows:
  "Error: Failed to insert/update event in database".
• cpsemd.elg shows:
  "ERROR: new row for relation "seam_event_XX" violates check constraint "cons_rowid"".
• cpsemd.elg shows (Windows Server 2008):
  

   CSeamApplication::_CloseLastPartition() - Failed to get num of rows from partition db
   CSeamApplication::_InitDBPartitions() - failed to close the last partition.
   Cannot create partitions in db
  

• All other processes are working fine.

Cause

SmartEvent server's virtual partition for events in the database became full, but did not automatically create a new partition within the database.

This happened due to a wrong database partition generation configuration file being used after SmartEvent was upgraded.

Solution

Note: This article is not relevant for NGSE. In NGSE, there is no SQL DB.

Note: The commands below are using a variable for CPshrd-R7x.xx. This needs to be replaced with the correlating path for YOUR version. DO NOT COPY and PASTE this command directly. For help on determining your version, via command line (expert mode) run #echo $CPDIR to see what CPshrd-Rxx resolves to.

Follow these steps on the involved SmartEvent server:

• On Gaia / SecurePlatform OS (in Expert mode):
  

  [Expert@HostName]# evstop
  [Expert@HostName]# /opt/CPshrd-R7x.xx/database/postgresql/bin/psql -U cp_postgres -p 18272 -f $RTDIR/conf/partition.sql events_db
  [Expert@HostName]# /opt/CPshrd-R7x.xx/database/postgresql/bin/psql -U cp_postgres -p 18272 events_db -c "select * from generate_new_partition(100000);"
  [Expert@HostName]# evstart

• On Windows OS (in Command Prompt):
  

  C:>\ evstop
  C:>\ cd /d "%CPDIR%\database\postgresql\bin\"
  C:\...\postgresql\bin> psql.exe -U cp_postgres -p 18272 -f "%RTDIR%"\conf\partition.sql events_db
  C:\...\postgresql\bin> psql.exe -p 18272 -c "select * from generate_new_partition(100000);" -U cp_postgres events_db
  C:>\ evstart