CPU at 100% – How do I connect to the box ?

Did you know you can setup a priority queue in your Check Point R77.30 appliance so that should the device become unresponsive due to high CPU load, you can still connect to it?

Have you ever had to run into a DC and pull a power cord because you could not connect to a box? Well now you can setup priority queues so you won’t have to do that.

Priority Queues are a mechanism that are intended to prioritize part of the traffic when we need to drop packets because the Security Gateway is stressed (CPU is fully utilized). In R77.20 and lower versions, when the CPU became fully utilized, part of the traffic was dropped regardless of the traffic type. As a result, control connections (described below) were dropped, which had serious negative impact (e.g., no SSH connectivity). In addition, several “heavy” connections could cause high CPU load on Security Gateway and cause issues for all other connections. However  R77.30 is “protecting” the CPU cores, on which Firewall is running.

To set this up follow theses instructions:

Instructions:

To check the current mode on Security Gateway:
[Expert@HostName]# fw ctl multik get_mode

To fully enable the Firewall Priority Queues on Security Gateway:

Note: In cluster environment, this procedure must be performed on all members of the cluster.
1.Run in Expert mode:
[Expert@HostName]# fw ctl multik set_mode 9

2.Reboot (in cluster, this might cause fail-over).

There are 3 modes (see chart)  and you can switch easily between them.

snip1
Firewall Priority Queues feature are now fully enabled however it is not currently on.  When is it on?  It turns on only in an extreme condition like when the CPU is overloaded.  The queues themselves are already predefined. See the chart below:

snip

You can also use this feature to monitor the Heavy Connections (that consume the most CPU resources) without interrupting the normal operation of Firewall, using the same command fw ctl multik set_mode 1

To learn more specifics check out sk105762

Happy uptime !!

references:

sk105762
sk105261
sk52421

 

Troubleshooting Identity Awareness

Domain administrator Credentials (Be sure to Use a Domain Administrator when hooking to Active Directory from the Wizard)

Security Gateway – Domain Controller communication

In order to configure and use AD Query (ADQ), the Security gateway must have connectivity to the Domain Controllers via DCE-RPC (port 135, and later a dynamic coordinated port), and LDAP / LDAP over SSL, according to your Domain Controller configuration. (Note: LDAP over SSL must be configured explicitly on your Domain Controllers).

Configuring the Firewall

If a Security Gateway is located between the Security Gateway with Identity Awareness/log server and the Active Directory controller, configure the Firewall to allow WMI traffic. If this is the case See To create Firewall rules for WMI traffic (below)

During the First Time Configuration Wizard. SmartDashboard – Domain Controller communication

In order for the wizard to be able to configure AD Query (ADQ), it must have connectivity to the Domain Controller. For this step, connectivity includes both TCP/IP connectivity (i.e., pings) and being able to perform DNS queries for it (i.e., running ‘nslookup’, ‘set type=srv’, ‘_ldap._tcp.your_domain.here’ succeeds). It is preferable to run the wizard from a computer that is a Domain Member, since then it can detect and configure all of the Domain Controllers. If you run it from a computer that is not a Domain Member, only one Domain Controller (that is entered manually) is being configured, and you will have to enter the rest of them manually. If you do not have connectivity when running the first time wizard, you will have to create an LDAP account unit manually for AD Query (ADQ) to work.

 

To verify if the WMI service is running on the domain controller:

Click Start > Run.

Enter services.msc in the Run window.

Find the Windows Management Instrumentation service and see that the service started.

If it did not start, right-click this service and select Start.

 

Use wbemtest to Verify WMI to verify that WMI is functional and accessible.

Click Start > Run.

Enter wbemtest.exe in the Run window.

In the Windows Management Instrumentation Tester window, click Connect.

In the Connect window, in the first field, enter the Domain controller, in this format: \\<IP address>\root\cimv2

In the Credentials > User field, enter the fully qualified AD user name. For example: ad.company.com\admin

Enter a password for the user.

Click Connect.

If the Windows Management Instrumentation Tester window re-appears with its buttons enabled, WMI is fully functional.

If the connection fails, or you get an error message, check for these conditions:

Connectivity problems

Incorrect domain administrator credentials.

WMI service is not running

A Firewall is blocking traffic between the Security Gateway with Identity Awareness/log server and domain controller.

 

To verify your domain administrator credentials:

Click Start > Run.

Enter \\<domain controller IP>\c$ in the Run window. For example: \\11.22.33.44\c$.

In the Logon window, enter your domain administrator user name and password.

If the domain controller root directory appears, this indicates that your domain administrator account has sufficient privileges. An error message may indicate that:

If the user does not have sufficient privileges, this indicates that he is not defined as a domain administrator. Obtain a domain administrator credentials.

Be Sure:

You entered the incorrect user name or password. Check and retry.

The domain controller IP is incorrect or you are experiencing connectivity issues.

Verify the WMI Service is running.

 

Confirm that Security Event Logs are Recorded

If you have checked connectivity but still do not see identity information in logs, make sure that the necessary event logs are being recorded to the Security Event Log.

AD Query reads these events from the Security Event log:

Windows 2003 servers: 672, 673, 674
Windows 2008 servers: 4624, 4768, 4769, 4770.
Windows 2012 servers: 4624, 4768, 4769, 4770

Make sure you see the applicable events in the Event Viewer on the domain controller (My computer > Manage > Event Viewer > Security). If they are not there however follow theses steps:
The Audit Policy is defined from the Group Policy Management editor.
1.Log on to Windows Domain Controller server with an account that has Administrator rights.
2.Make sure that the Group Policy snap-in is installed.
3.Open the Group Policy Management Console (GPMC).
4.Navigate to “Default Domain Controller’s Policy”:
Group Policy Management Console -> Domain Controllers -> Default Domain Controllers Policy
5.Right-click on the ‘Default Domain Controllers Policy’ and click on “Edit”.
6.From the Group Policy Management Editor, navigate to “Audit Policy” node:
Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> Audit Policy.
7.From the right pane, double-click the policy that you want to configure (enable/disable):
8.Configure:

Group_Policy_Management

◦”Audit account logon events” – select both “Success” and “Failure”.
◦”Audit account management” – select both “Success” and “Failure”.
◦”Audit directory service access” – select “Success” (for GPO and OU Auditing).
◦”Audit logon events” – select both “Success” and “Failure” (for Local Logon auditing).

To create Firewall rules for WMI traffic:

In SmartDashboard > Firewall, create a rule that allows ALL_DCE_RPC traffic:

Source = Security Gateways that run AD Query

Destination = Domain Controllers

Service = ALL_DCE_RPC

Action = Accept

Save the policy and install it on Security Gateways.

Note – If there are connectivity issues on DCE RPC traffic after this policy is installed, see sk37453 for a solution.

For an in depth look as to how AD query and WMI work look at sk60301

 

References:

 

sk60301

sk37453

sk99006

R77 Identity Awareness R77 Versions Administration Guide

Setting the Cluster_id from the command line on Check Point

This is the replacement for the mac-magic as of R77.30. If you are using an older version you still have to use mac-magic. We now use the cluster id. The reason that we set a cluster id is because if you wish to put two or more clusters on the same subnet this creates a problem. The Cluster Control Protocol (CCP) packets that are sent between the members of the same cluster, reach the neighbor cluster (connected to the same network) and “confuse” it.

So by changing this number ensures communication to the correct member.

Most of time you are setting this when you are running the first time wizard in the WebUI, however should you need to change it, or establish the cluster id from the command line it is a simple setup.

This must be done on both cluster members.

  1. Login as Expert
  2. View the current setting type: cphaconf cluster_id get
  3. type: cphaconf cluster_id set <number to set to ex…1-252>
  4. reboot the appliance
  • It is very important that you do not use 253, 254. These are the Default cluster id’s of the device. So if you are putting another cluster on the wire and you change the id to say 254 you haven’t achieved anything as that is the default.  you will still be getting errors in the log.

This command sets the value of Cluster Global ID permanently – the configured value is automatically and immediately inserted into the$FW_BOOT_DIR/ha_boot.conf file

You can also do this in GAIA when you first install the box. Here is a screenshot from the first time wizard.

cluster-id-set

references:

SK25977

You’re watching TV – Is it also watching you?

Check Point Software recently mentions this on their blog site regarding EZCast. (See the full post here: http://blog.checkpoint.com/ )

“It’s an HDMI dongle-based TV streamer that converts your regular TV into a smart TV and allows you to connect to the Internet and other media.”

“Since the EZCast dongle runs on its own Wi-Fi network, entering the network is actually quite easy. This network is secured only by an 8-digit numeric password, which can be easily cracked.”

Check Point discusses the potential of information leakage that can come once a brute force attack (which they successfully did) is executed.

They further go on to ask the question “Would you sell access to your network for $25 dollars? Because that’s what you’re essentially doing when you buy and use this device.”

Since there are roughly 5 million users and EZCast has not bothered to address this, all I can say is enjoy that movie marathon.

Muhahahahahahahaha !!!!

 

References:

http://blog.checkpoint.com/

SPAN port on a Switch

Here is a quick post on how to configure a SPAN port on some of the various switch gear should you need one

term

 

 

 

Cisco Catalyst 2850, 2940, 2950, 2955, 2960, 2970, 3550, 3560, 
3560-E, 3750, 3750-E 4500/4000

conf t
monitor session 1 source interface gigabitEthernet 0/17 both
monitor session 1 destination interface gigabitEthernet 0/15
exit
write mem

C6500/6000 Series Switches That 
Run Cisco  IOS System Software, Cisco Nexus Series Switches That Runs  
NX-OS Software

Syntax:
monitor session session_number source interface interface-id [, | -] [both | rx | tx]
monitor session session_number destination interface interface-id

Cisco Catalyst 2900, 4500/4000, 5500/5000, 
  and 6500/6000 Series Switches That Run CatOS

Syntax:
set span source_port destination_port [rx | tx | both]

Juniper

root@switch# edit
root@switch# set ethernet-switching-options analyzer mirror-3d input egress interface ge-0/0/6.0
root@switch# set ethernet-switching-options analyzer mirror-3d input ingress interface ge-0/0/6.0
root@switch# set ethernet-switching-options analyzer mirror-3d output interface ge-0/0/13.0
root@switch# commit

Brocade

Monitoring an Individual Trunk Port
By default, when you monitor the primary port in a trunk group, aggregated traffic for all the ports in the trunk group is copied to the mirror port. You can configure the device to monitor individual ports in a trunk group. You can monitor the primary port or a secondary port individually.

To monitor traffic on an individual port in a trunk group, enter commands such as the following:

ServerIron(config)# mirror ethernet 2/1
ServerIron(config)# trunk switch ethernet 4/1 to 4/8
ServerIron(config-trunk-4/1-4/8)# config-trunk-ind
ServerIron(config-trunk-4/1-4/8)# monitor ethe-port-monitored 4/5 ethernet 2/1 in

No Downtime Check Point Cluster Upgrades

Ever wonder how to do a no downtime upgrade on a Check Point Cluster?

Many of my customers have asked me for best practices for upgrading their clusters. I have shown the methods (and done them) throughout the years of failover back and forth between members using state to effectively create a no downtime scenario, however there was no real guarantee of uptime or that a connection wouldn’t be dropped.

Well now in R77 you can do what is called a full connectivity upgrade. Basically it means:

• Connection failover is guaranteed.
• There is always at least one active cluster member that handles the traffic.
• Connections are synchronized between cluster members running different Check Point software versions.

Sound good ???

Here are none simple steps on how you do it:

Before you upgrade:
• Make sure that the cluster has 2 members, where one of them is the Active member and the other member is in Standby.
• Get the sync interface IP address and the cluster member ID of the Active cluster member.

From the command line in Expert type: cphaprob stat

To upgrade the cluster:

  1. In SmartDashboard:
    – In the Gateway Cluster General Properties window, change the Cluster version to the upgraded one.
    – In the Install Policy window, go to the Installation Mode area > Install on each selected gateway independently and clear For Gateway Clusters install on all the members, if it fails do not install at all.
  2. Install the security policy on the cluster.
    (Note – The policy successfully installs on the Ready cluster member and fails to install on the Active cluster member. This is expected, ignore the warning.)
  3. On the Active cluster member, run: cphaprob stat
    (Make sure the state is Active or Active Attention, and record the Sync IP and the Member ID of the cluster member.)
  4. On the upgraded cluster member, run these commands:
    cphaprob stat
    (Make sure that the cluster member is in Ready state.)
    cphacu start
    (The Connectivity Upgrade runs. When it finishes, the member’s state is Ready for Failover.)
    cphacu stat
    (Make sure that the Active cluster member handles the traffic.)
  5. On the Active cluster member, run these commands:
    cphaprob stat
    (Make sure the local member is in Active or Active Attention state, and the upgraded member is in Down state.)
    cpstop
    (The connections fail over to the upgraded cluster member.)
  6. On the upgraded cluster member, run: cphaprob stat
    (Make sure that it is now in the Active state.)
  7. On the new upgraded cluster member, run: cphacu stat
    (Make sure it handles the traffic.)
  8. Upgrade the former Active cluster member.
    (Make sure to reboot it after the upgrade.)
  9. Install Policy.

Note: After the cluster upgrade is complete, the Cluster Control Protocol works in broadcast mode.  I recommend this mode however if you were running in multicast mode and wish to return to that state then on all cluster members run the command: cphaconf set_ccp multicast

That is all it takes to have a successful full connectivity upgrade of an HA cluster. If however you have more than a two member cluster or have a VSX cluster or would like more information check out the CP_CU_BestPractices.pdf guide on how to do all this.

Capture

References :

Connectivity Upgrades Best Practices R77 (CP_CU_BestPractices.pdf)

 

Backing up your Check Point Infrastructure

I have had many a customer ask me “…how should I backup my Check Point?”

There have actually been many posts in the usercenter about this showing how and recommended ways. However I decided to consolidate some of that and put them here in this post so that you can read it. I am also including the specific articles used that you can reference.

Bottom line if you are backing up your stuff that is good news, however if you are not then you need to be doing so!!

Here are two separate scenarios on how to backup and restore your Check Point Infrastructure you can use. I will also include a supplemental backup that you can do for your management station as well (see Management Supplement #1)

BACKUP SCENARIO #1:  SNAPSHOT

The snapshot creates a binary image of the entire root disk partition. This includes Check Point products, configuration, and operating system. It is important to note that this does not include the log files as logs are not stored in the root partition. There are still space requirements you should consider before doing scenario #1.

To create the snapshot image requires free space on the Backup partition. The required free disk space is the actual size of the root partition, multiplied by 1.15.

 The free space required in the export file storage location is the size of the snapshot multiplied by two.

 The minimum size of a snapshot is 2.5G, so the minimum free space you need in the export file storage location is 5G.

To create a snapshot image:

It is important to note you can do this from the command line or the web portal (WebUI) (see GAIA Administration Guide for the command line), so just from a web browser login to the GAIA portal.

1. In the tree view, click Maintenance > Image Management.

2. Below available images, click New Image. The Create New Image window opens.

3. In the Name field, enter a name for the image.

4. Optional: In the Description field, enter a description for the image.

5. Click OK.

You can choose to leave it on the filesystem, &/or you can save it off.  However, if you save it off as well as leave the latest and greatest on the filesystem you will have access to it in a pinch.

snap3

 

How to Restore an image you took but don’t have it on the filesystem:

Now that you have your image on the filesystem (or it was already there) here is how to restore it.

1. In the tree view, click Maintenance > Image Management.

2. Select an image.

3. Click Revert. The Revert window opens.

Of course this procedure means the device is in a down state at the time, however if you are having to revert to an image you created I am guessing that is the least of your worries at that moment. But worry not!  Restore it and you will be up and running with the system looking like it was when you took the snapshot.

BACKUP SCENARIO #2:  SYSTEM BACKUP

System Backup can be used to backup current system configuration. A backup creates a compressed file that contains the Check Point configuration including the networking and operating system parameters, such as routing and interface configuration etc., but unlike a snapshot, it does not include the operating system, product binaries, and hotfixes.

You can store backups locally, or remotely to a TFTP, SCP or FTP server. You can run the backup manually, or do a scheduled backup

To add a manual one time backup:

1. In the tree view, click Maintenance > System Backup

2. Click Add Backup.

The New Backup window opens.

3. Select the location of the backup file:

This appliance

TFTP server. Specify the IP address.

SCP server. Specify the IP address, user name and password.

FTP server. Specify the IP address, user name and password.

To add a scheduled backup:

1. In the tree view, click Maintenance > System Backup.

2. Click Add Scheduled Backup. The New Scheduled Backup window opens.

3. In Backup Name, enter the name of the job. Use alphanumeric characters only, and no spaces.

4. In Backup Type, enter the location of the backup file.

    This appliance

    TFTP server. Specify the IP address.

    SCP server. Specify the IP address, user name and password.

    FTP server. Specify the IP address, user name and password.

5. In Backup Schedule, select the frequency (Daily, Weekly, Monthly) for this backup. Where relevant, enter the Time of day for the job, in the 24 hour clock format.

6. Click Add. The scheduled backup shows in the Scheduled Backups table.

back2

To restore from a backup:

1. In the tree view, click Maintenance > System Backup.

2. Select the backup file and click Restore Backup.

 

MANAGEMENT SUPPLEMENT #1

To do a migrate export for your management station (SmartCenter) from the command line.

This migrate export captures your management station’s database (policies, objects, certs etc…) however it is Not capturing any O/S related info such as the IP, hostname, routing etc…

1. Log in to the expert mode.

2. Type: cd $FWDIR/bin/upgrade_tools

3. Type: ./migrate export <exported database name>.tgz

4. Do the instructions shown on the screen. This creates the <exported database name>.tgz file.

5. Copy the file off and save it.

That is all folks. Not that hard to do, and once you are setup you won’t need to worry should anything ever happen.  I recommend you do all of the scenario’s I mentioned on a regular basis (just like any server backup) then you can truly be successful.

You can read more information about what I wrote here at our usercenter.

References:

sk108902

Gaia Administration Guide R77

Gaia R77 Installation and Upgrade Guide