TRAINING‎ > ‎RESOURCES‎ > ‎

Check Point/SPLAT/Network Debug Cheat Sheet

Unix Cheats
 
Unix Cheats
ifconfig List all network interfaces - look for collisions
ifconfig | fgrep inet Short list of network IPs

ifconfig | grep -i 'hwa\|inet'

Short list of IPs, ethernet interfaces, and IPs
route -n Print routing table with no DNS resolution
arp -an Print out arp table
ls -alth List 'a'll files in 't'ime order with 'h'uman readable sizes

more <filename>

<somecommand> | more

The more command will print out a file and scroll it so you can read it at your pace.

/<searchtext> : search forward for <searchtext>
<cr> : read one more line
<space> : read one page
q : quit
? : help

ls -alt | tee /tmp/output.txt List the directory, the feed it to "tee" which lists on display AND feeds into file at same time
find . -follow | xargs ls -ald | more List all files in directory, follow symbolic links, and print the file modification time "ls -ald"
fgrep "search for" * search for "search for" in inside a file in the current directory
find . -follow -mmin -10 | xargs ls -ald | more Search for files modified less than 10 minutes ago and print mod dates. Follow all symbolic links
Filenames: find . -follow -mmin -10 | fgrep "lookforfilename" Looking for a filename that has been modified in last 10 minutes
Filecontent: find . -follow -mmin -10 | xargs fgrep -i "lookforcontent" | more Search for files modified less than 10 minutes ago and search inside files for content (not the filename!) "lookforcontent"

find / -follow -type f -size +10240k

find / -follow -type f -size +10240k -size -102400k 

Search for files larger than +10MB. -10240k is less than -10MB

File between +10MB and -100MB

command > filename send standard output to "filename" (by default error output goes to terminal)
command 2>&1 | more send error output "2" to(>) same output as standard output "&1"....which is the display. Pipe all through more
command 2> /bin/null | more Send all the error output to null bucket so output just has clean data in it
CTRL-v CTRL-o ENTER Resets terminal to text in case binary data changes display
stty rows 80 columns 100 Resets terminal size to window of 80 by 100
dmesg | more Look at all the boot messages, usually used to find debug errors
Linux: traceroute -n
Windows: tracert -d
Traceroute --- Turn off DNS!

dig @<server IP> <domain name>

dig @172.17.1.2 www.google.com

Direct query to DNS server @ IP 172.17.1.2
netstat -r     routing tables
 netstat -an  Port numbers that are being used or listening on
 ps -aux, ps -auxwf  print out all processes or with subprocess tree list.

 top

<space bar>

F -> k

F ->n

f -> X

R    

 Realtime display of processes and mem utilization

refresh

 Sort on CPU

Sort on MEM

Sort on Command name

Toggle reverse sort on/off

 tftp <ip of tftp server>

binary

put <filename>

 

ethtool <ethX e.g. eth0>

ethtool -p eth0

ethtool -i eth0

 Display physical network card information (limited on VMware)

Blink the LED on eth0 port

Get driver version info

 
Smart Gateway Debug Commands
clock bios time and date
cpconfig change SIC, licenses and more
cplic print license information
cpstart;cpstop start all checkpoint services
cpstat fw show policy name, policy install time and interface table
cpstat ha high availability state
cpstat os -f cpu checkpoint cpu status
cpstat os -f routing checkpoint routing table
cpstop stop all checkpoint services
cpwd_admin monitor_list list processes actively monitored. Firewall should contain cpd and vpnd.
expert change from the initial administrator privilege to advanced privilege
fw ctl iflist show interface names
/bin/date OS time and date
fwm load <policyname> <gatewayobjectname> On smartcenter or MDM, verify and compile and load the policyname onto the targetgateway. When it compiles, it creates a file called  $FWDIR/conf/<policyname.pf>, this is the compiled inspect script. From that it generates $FWDIR/conf/<gateway>/rulebases_5_0.fws which is loaded into gateway.

fw fetch 10.0.0.42

get the policy from the firewall manager (use this only if there are problems on the firewall). Downloads rulebases_5_0.fws from 10.0.0.42. This has latest policy in it

fw unloadlocal Remove all policy and security enforcement from SPLAT. Make it a straight linux box basically
sysctl -w net.ipv4.ip_forward=1 After unloading policy, make SPLAT route through the box. Turn on forwarding
fw ver -k Firewall version and kernel version
cpinfo Prints out TONS of FW debug information for help desk
fw stat firewall status, should contain the name of the policy and the relevant interfaces, i.e. Standard_5_1_1_1_1 [>eth4] [<eth4] [<eth5] [>eth0.900] [<eth0.900]
fw stat -l show which policy is associated with which interface and package drop, accept and reject

fw tab

fw tab | grep '\--' | more

displays firewall hash tables. Note these are tables that are reserved for firewall kernel hash tables. fw ctl mem

Dump out names of tables stored in hash memory ' fw ctl pstat' (hmem)

fw tab -s -t connections number of connections in state table
fw tab -t xlate -x clear all translated entries (emergency only)
fwm lock_admin -h unlock a user account after repeated failed log in attempts
fwm ver firewall manager version (on SmartCenter)
sysconfig configure date/time, network, dns, ntp
upgrade_import run ‘/opt/CPsuite-R65/fw1/bin/upgrade_tools/upgrade_import’ after a system upgrade to import the old license and system information.
hwclock show the hardware clock. If the hardware and operating system clocks are off by more than a minute, sync the hardware clock to the OS with "hwclock –systohc"
cpd_sched_config print Print out CP batch queue -- CP version of crontab
fw ctl arp List all the proxy arp entries for manual arp brought into the kernel from $FWDIR/conf/local.arp. Also have to check Global Properties:NAT:Merge manual arps

======NAT DEBUG=========

fw ctl debug -buf 32000
fw ctl debug + xlate xltrc nat
fw ctl kdebug -f > /tmp/nat.out
fw ctl debug 0

--Reserve space
-- Turn on NAT debug
- Dump to file
- Turn off NAT debug

fw monitor -i -p all > outputascii.txt
  

-o output.cap  

-x 0

-e "accept ip_src=1.2.3.4"

 Dump traffic through iIoO stacks and ouput to ascii. -i flushes buffers immediately so you get all the output written to the output file.

Dump traffic through iIoO stacks and output to Wireshark format for export.

Packet data dump starting at offset 0

Only filter packets from 1.2.3.4 [NOTE:$FWDIR/lib/tcpip.def has shortcuts for filtering -- ip_src is one example of a shortcut/macro]

                     Common VI commands
 vi <filename>  

 :q!

ZZ

 quit-no save

quit: save data

 <arrow keys>

<pg up><pg down>

 up/down/sideways

 x

dd

 delete character

delete line

/someword

?someword

n

N

 search forward for someword

search backwards for someword

search for next instance

search for previous instance

 i

a

<ESC>

u

 insert character at current spot

append character after current spot

escape insert mode

undo last change

 o

O

<ESC>

u

Create line above current line and start inserting characters

Create line above current line and start typing characters

escape insert mode

undo last change

 
Unix/Splat Performance Queries
ifconfig List all network interfaces - look for collisions and errors on interfaces
 ps auxwf    Print out process hierarchy with cpu times, and full commands that process is executing  
cpstat os -f all Best overall view of OS with descriptions. Look for FREE space to make sure there is free space. Look at Queue lenghts to see if things are backing up.
Look for % CPU time.
fw ctl pstat

show control kernel memory and connections:

Hash Kernel Memory: Total Memory Blocks Used/Unused/%: State table memory--->Make sure this has free memory
System Kernel Memory: Allocations: Free: Application Memory -> Make sure there is free memory for applications
Kernel Memory: Free: Firewall kernel memory-> Make sure there is free memory
Connections: Peak concurrent vs concurrent: Make sure under default config 25000
top Dynamic list of processes and the resources they utilize


vmstat

vmstat 5


Memory and virtual memory usage

Display every 5 seconds

Look for :

w: Number of processes blocked waiting for resources to run. Should be low number.
free: The amount of idol memory available for swapping. Should be big number.
so: swap out: means running out of physical memory so start "swap out". Should be
low number and decreasing
bi: blocks read from disk - Compare to nonbusy device, should be steady number
bo: blocks written to disk - Compare to nonbusy device, should be steady number
cs: context switches - Number of times a process goes from idle to running. Takes a
lot of CPU and swap. Should be low or steady number

us: cpu time spent running user code
sy: cpu time spent running kernel code
id: Time spent ideal
wa: Time spent waiting for IO to happen - Watch this one
df -h Disk usage of all the drives
netstat -i packets dropped/errors per interface
 Logging Commands
fw log -c drop Entries in the log 'drop' column. also can use 'accept' and 'reject'
fwm logexport -i <log name> -o <outputfilename export an old log file on the firewall manager
fw log -n <fwlogname.log> > /tmp/logout.txt dump logs into file...do not use DNS resolution -n, much faster! NO DNS! -n
fw log -n < fwlogname.log> | tee /tmp/logout.txt dump log files both to display AND to file logout.txt. NO DNS! -n
fw logswitch rotate logs, clear out current log and archive it based on date
fw lslogs list firewall logs
fw log -f tail the current log
fw log -n <logfile>.log
log list
show all logs kept by gateway/mgr. Both *.log, *elg,
log show #
dump a log listed by 'log list'
fwm logexport -i <log name> -o <output name.txt> NOTE: 'fwm' not 'fw'. Export ogs to ascii to output_name.txt file
fw log -b <MMM DD, YYYY HH:MM:SS> <MMM DD, YYYY HH:MM:SS> search the current log for activity between specific times, eg

fw log -b "Jul 23, 2009 15:01:30" "Jul 23,2009 15:15:00"

tcpdump

tcpdump -i <interface> -n -s 500 -w <outputfile> -X [command]

eg. tcpdump -i eth1 -n -w /tmp/netout.cap -X [command]

eg. tcpdump -i eth1 -n | tee /tmp/netout.cap --- dump to file and screen

  • -i <interface> commonly eth0/1/2/3
  • -n no DNS resolution, faster
  • -s 500 size of data packet
  • -w output file, can be used to feed into Wireshark
  • -X ascii and hex output
Expression Modifiers
  • ! or not
  • && - and
  • || - or
  • NOTE: If you use (),||,&&,! then enclose the whole command
    in quotes ' host 1.1.1.1 && host 2.2.2.2' because the shell
    will use the && before tcpdump does.

Common Command Operators

  • [ether] [src|dst] host <host> | net <net>/len
  • [tcp|udp] [src|dst] port <p1> | portrange <port> - <port>
  • [src|dst] net <netip>/mask
  • arp
  • icmp
  • proto

Examples commands:

  1. host 1.1.1.1
  2. src net 10.1.1.0/24
  3. src host 1.1.1.1 or (dst host 3.2.1.4 and src port 53)
  4. ether src host 00:0c:29:80:11:0c -- monitor all data packets from MAC address
  5. ether src host 00:0c:29:80:11:0c and arp -- monitor arp packets from MAC address
  6. port 22 or arp ---- figure out why SSH is not connecting .... probably because ARPs are not being returned.


Wireshark

Expression Modifiers:

  • a == b
  • a != b
  • a and b
  • a or b
  • !(a or b)

Common filters

ARP

  • arp --- filter just arps
  • arp or icmp --- filter arps and icmp
  • icmp --- filter just icmp

    IP
  • ip.host == 1.1.1.1
  • ip.src_host == 1.1.1.1
  • ip.dst_host == 1.1.1.1

    TCP
  • tcp.port == 22
  • tcp.dstport == 22
  • tcp.srcport == 22

Complex Filter Examples:

  • arp or ip.host == 1.1.1.1
  • arp or icmp and !(ip.host == 1.1.1.1)
  • NOTE: use !(ip.host == 1.1.1.1) and NOT ip.host != 1.1.1.1
  •  
 
 
License Commands
 
 

cplic print -s

Print out all licenses with signatures

cplic del XXXX

Delete license with signature XXXX 

 cplic print -x | awk '{print $3}' | xargs -n cplic del

 Delete all licenses. You will get errors but in end it will clear them all out.

cp contract put <file>

cplic put -l <file>

 Install a license file or a contract file. Licenses are for software, contracts are for subscriptions like IPS, spam, URL filtering

 cplic print -p

Print out the detailed licensing info after being translated by $CPDIR/conf/cp.macro 

 
 
 
 
 
 
 Splat Gateway Filesystem 
 $CPDIR/log
cpd logs, setup logs, general check point product logs
$FWDIR/database && $FWDIR/state Where policy is installed
$FWDIR/log/ Directory of log files. Use "ls -alt" to find recently modified log files
/var/log/messages Linux OS logs
$FWDIR/log/*.elg component text log files
$FWDIR/log/fw.log log file that shows up in Smart Tracker
$FWDIR/conf  FW configuration files
Smart Gateway HA debug Commands
cphaprob ldstat display sync serialization statistics
cphaprob stat list the state of the high availability cluster members. Should show active and standby devices.
cphaprob syncstat display sync transport layer statistics
cphastop stop a cluster member from passing traffic. Stops synchronization. (emergency only)
cphaprob -a if Display state of interfaces
cphaprob -ia list List all the monitored devices and their status to figure out why the firewall failed over.
clusterXL_admin up|down fail over device. NOTE you have to failover other device to get this device back to active, not automatically flips back to highest priority. Unless you set autorecovery in clusterXL menu for

 Smart Gateway VPN debug Commands
vpn tu list and kill tunnels

fw ctl chain

fw monitor -p > /tmp/outputfile.txt

Watch iIoO stack traffic and data being decrypted

fw ctl debug -buf 32000
fw ctl debug -m VPN all
fw ctl kdebug -f > /tmp/vpn.out
fw ctl debug 0       ----- Turn off!!!!

Turn on VPN kernel debug and send output to file /tmp/vpn.out

Turn off after done!!!

vpn debug trunc

vpn debug off      ---- Turn off!!!

 Debug the setting up key exchanges and tunnel testing. Output is in $FWDIR/log/vpn.elg and ike.elg


Smart Center Filesystem
$FWDIR/conf
$FWDIR/log

<year>-<month>-<day>_<time>.log

eg. 2011-09-02_105546.log - Name of logfile switched on 9/2/2011

Names of log files when you execute a fw logswitch
/var/log/messages
$FWDIR/conf/Standard.pf Rulebase saved by Smart Dashboard
$FWDIR/conf/rulebases_5_0.fws Compiled rule bases pushed to gateway
 $CPDIR/log  cpd daemon and intercommunications logs  

MDS file system

$MDSDIR/log

MDS logs

$MDSDIR/conf

MDS global databases

cd $FWDIR

current DMS, make sure you do a mdsenv

MDS command lines
mdsstat List all the DMS and their statuses
mdsenv set MDS environment to a specific domain (listed in mdsstat)
mcd change environment to domain specified in mdsenv
mdsstop Stop all of MDS
mdsstart Start all of MDS
mdsstop_customer customer start a single DMS
mdsstart_customer customer stop a single DMS
mdscmd command line version of the SDM GUI

mdscmd runcrossdomainquery -all query_rulebase -n G_MDS

Search all DM's rulebases dor G_MDS

mdscmd runcrossdomainquery -all query_network_obj -c dms

Search all DMS's object files for partial name "dms"
Comments