Cisco Firepower 2100 Series Forensic Investigation Procedures for First Responders


Introduction

Prerequisites

Step One - Cisco Firepower Device Problem Description

Step Two - Document the Cisco Firepower Runtime Environment

Step Three - Verify the Integrity of System Files

Step Four - Verify Digitally Signed Image Authenticity

Step Five - Verify FTD Memory .text Segment Integrity

Step Six - Cisco Firepower Crashinfo File/Core File

Step Seven - ROMMON Settings Check

Related Documentation

Cisco Firepower 2100 Series Forensic Report Checklist

Revision History




Introduction

This document provides steps to collect forensic information from Cisco Firepower 2100 series appliances running Firepower eXtensible Operating System (FXOS) Software when compromise or tampering is suspected. It outlines a number of commands that can be run to gather evidence for an investigation along with the respective output that should be collected upon running these commands. This document also provides information on how to perform integrity checks on FXOS system and Firepower Threat Defense (FTD) application images and includes a procedure for collecting a memory dump, crashinfo file, and a core file from a Cisco Firepower device.

Note: Firepower Threat Defense (FTD) investigation procedures for the ASA 5500-X series of firewalls are covered in a separate publication, which can be accessed here: Cisco Firepower Threat Defense Forensic Investigation Procedures for First Responders.

IMPORTANT: DO NOT REBOOT THE DEVICE. Rebooting a device during the initial stage of an assessment will irrecoverably lose all volatile information contained within the device. (e.g. RAM contents, arp & routing tables, NAT translations, ACL hit and drop counts, etc.)

Note: It is highly recommended that a device suspected of tampering or compromise be isolated from the network prior to conducting an initial forensic examination. This may prevent remote unloading of any implants or malware installed on the device and will prevent an adversary from monitoring commands entered on the device under investigation.

If you require assistance or have questions regarding the following procedures, contact the Product Security Incident Response Team (PSIRT)

The main section of this document contains seven sections:

1.      Cisco Firepower Device Problem Description – Describes why the platform is a candidate for forensic examination

2.      Cisco Firepower Runtime Environment – Collects platform configuration and runtime state

3.      Verify the Integrity of FTD System Files – Examines system file image hashes for inconsistencies

4.      Digitally Signed Image Verification – Examines the FXOS operating system for proper signing characteristics

5.      Verify Memory .text Segment – Retrieves and calculates a hash of the FTD .text segment

6.      Crashinfo/Core File – Obtains crashinfo and core files from the running FTD application

7.      ROM Monitor Variables – Examines ROM Monitor settings for remote system image loading


Prerequisites

The procedures outlined in this document assume that the reader has a basic understanding of Cisco FXOS Software, Cisco Firepower Threat Defense Software, and Linux command syntax.

A valid cisco.com account is required to view individual FXOS Software and FTD firmware file hashes for software file integrity checking. For customers without a cisco.com account, a publicly available comprehensive list of file hashes (Bulk Hash File) can be downloaded from: https://www.cisco.com/c/en/us/about/trust-center/downloads.html


Step One – Cisco Firepower Device Problem Description

Describe in as much detail as possible WHY the device is a candidate for forensic examination. Are there configuration changes that cannot be explained? Is there unusual traffic originating from or terminating on the device? Are there anomalous entries in the device logs or in syslog messages? Is the device exhibiting odd behavior that cannot be attributed to a misconfiguration or a software or hardware defect?

Submit the problem description collected in this section to the relevant TAC SR and proceed to the next section of this document.


Step Two – Document the Cisco Firepower Runtime Environment

Note: The Cisco Firepower series of appliances is capable of running either Cisco Firepower Threat Defense (FTD) Software or Cisco Adaptive Security Appliance (ASA) Software under the FXOS operating system. The examples provided in this guide use commands and syntax suitable for FTD Software. Please see the Cisco ASA Forensic Investigation Procedures for First Responders guide for examples specific to Cisco ASA Software.

The initial stage of forensic information gathering is completed by issuing a show tech-support command and a number of optional show commands to collect more granular operating environment details. These commands are to be executed from the privileged EXEC mode of the FTD diagnostic CLI and some of the output produced may vary depending on the particular FTD Software version and/or features supported/configured on the device.

Note: For the Cisco 2100 Series of Firepower appliances, the default command prompt will vary depending on the method used to access the platform. If using SSH, the user will be placed in the FTD CLI. If using a console connection, the user will be placed in the FXOS CLI. Issue the connect ftd command to access the FTD CLI from the FXOS CLI.

Execute the following command from the FTD CLI prompt:

system support diagnostic-cli

Execute each of the following commands in the diagnostic CLI and record the output:


enable
terminal pager 0
show tech-support detail
dir all-filesystems

The following list of commands should be executed to gather additional information relevant to the current operating state of the device. Although the output of these commands is not required to perform a forensic analysis of an FTD platform, it may provide additional information regarding any unauthorized changes made to the device if compromise is suspected.

Execution of the following commands in this section of Step 2 is optional.


terminal pager 0
show history
show clock detail
show startup-config
show reload
show processes
show kernel process detail
show kernel ifconfig
show kernel module
show logging
show route
show eigrp nei
show ospf nei
show bgp summary
show arp
show ip address
show interface ip brief
show nat detail
show snmp-server user
show snmp-server group
show ipv6 interface brief
show ipv6 route
show conn all
show xlate
show aaa login-history

Submit all command output collected in this section to the relevant TAC SR and proceed to the next section of this document.


Step Three – Verify the Integrity of System Files

Connect to the FTD CLI. This can be accomplished from the FXOS CLI by issuing the connect ftd command.

From the FTD CLI, issue the following commands to assume root permissions, set the appropriate environment variables, run the system file integrity checks, and collect the necessary files for forensic assessment.

Access expert mode and sudo to the root account:


expert
sudo su -

Set the FIPS environment variable and run the integrity checks:


export FIPS_MODE=1
find /ngfw/var/sf/.icdb/* -name *.icdb.RELEASE.tar | xargs sha512sum
cat /proc/*/smaps > /tmp/all-process-smaps.txt
verify_file_integ.sh

Locate and retrieve copies of the following files:


verify_file_integ.sh
verify_signed_db.sh
db_manage.sh
/ngfw/etc/certs/*.crt
/ngfw/var/log/sf/verify_file_integ.log
/ngfw/var/tmp/merged-db/master.db

Create an archive of the files listed above and copy the archive off the platform:


tar -cvf SR-<sr_number>.tar
sha512sum SR-<sr_number>.tar
ftp or scp

An example of this procedure follows:


firepower# connect ftd
> expert
*************************************************************
NOTICE - Shell access will be deprecated in future releases
         and will be replaced with a separate expert mode CLI.
**************************************************************
admin@firepower:/$ sudo su -

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Password: 
root@firepower:~# export FIPS_MODE=1

# Check to see if the environment variable is set correctly
root@firepower:~# set | grep FIPS_MODE
FIPS_MODE=1

root@firepower:~# find /ngfw/var/sf/.icdb/* -name *.icdb.RELEASE.tar | xargs sha512sum
04f4f3cd8fab83099faf41eda608e7fd3164bd75f0505585d70f43a5cd70c3f8f11c338fab727904011c9c9de8fedd4f3f5ca34d49f1e370543dc04d97355d4e
/ngfw/var/sf/.icdb/0000/base-octeon-6.4.0-102.icdb.RELEASE.tar
63c99b2b92895188f921bfbde6f000d670b65ac07642375b9d9ed8aedb63761441241006a86afb047f7e27d02c10b4df4df8c860b8fba601f91b3a36bebf2cd8
/ngfw/var/sf/.icdb/0000/base-6.4.0-102.icdb.RELEASE.tar
fd5be05b711a5e13a26bdf7a6e9d8b43c8633cde9357f5c429e0b3f1fa4ba4acd203ad7f7bf3a2ec14be1214e75713df4fc2a9def6ac09b301ad2618c475c6e3
/ngfw/var/sf/.icdb/0001/patch-v6.4.0-OS-6.4.0.4.icdb.RELEASE.tar
720ea233cff7e59502e95e897a985f7832f992e7961541bc0fae0b5ca7b535479546ccbd0566db264d7ecc4cbc5c60a8df98894beb8c441d313458a8b2fd3bd9
/ngfw/var/sf/.icdb/0002/onboxui-6.4.0.4.icdb.RELEASE.tar
d63661140c552ea67e0841bc46e968b732d63932fc6b1bde6d528df8b7e8f55cdd0fc05a0d79b9bed6f10139be6ee7aaa6c65b383944dd1b9880ae8596cc966a
/ngfw/var/sf/.icdb/0002/patch-v6.4.0-6.4.0.4.icdb.RELEASE.tar # # Note: the file names, number of files, and hash values may vary depending on # the version of software running on the appliance and whether any software # updates have been applied. It is extremely important that the output # (including file names and hashes) generated by the command above be # submitted to the TAC SR. # root@firepower:~# cat /proc/*/smaps > /tmp/all-process-smaps.txt
root@firepower:~# verify_file_integ.sh
Running file integrity checks...
Successfully verified file integrity
# Once the file integrity verification script has completed, unset the # FIPS_MODE environment variable:
root@firepower:~# unset FIPS_MODE
# Identify the location of the system integrity scripts with the “which” # command:
root@firepower:~# which verify_file_integ.sh
/ngfw/usr/local/sf/bin/verify_file_integ.sh
root@firepower:~# which verify_signed_db.sh
/ngfw/usr/local/sf/bin/verify_signed_db.sh
root@firepower:~# which db_manage.sh
/ngfw/usr/local/sf/bin/db_manage.sh
# Archive a copy of these scripts, along with all certificates found in # /ngfw/etc/certs/, and the files
# /ngfw/var/log/sf/verify_file_integ.log and # /ngfw/var/tmp/merged-db/master.db
root@firepower:~# tar -cvf SR-1234567890.tar /ngfw/usr/local/sf/bin/verify_file_integ.sh /ngfw/usr/local/sf/bin/verify_signed_db.sh
/ngfw/usr/local/sf/bin/db_manage.sh
/ngfw/etc/certs/*.crt /ngfw/var/log/sf/verify_file_integ.log /ngfw/var/tmp/merged-db/master.db
/tmp/all-process-smaps.txt
tar: Removing leading `/' from member names
/ngfw/usr/local/sf/bin/verify_file_integ.sh
/ngfw/usr/local/sf/bin/verify_signed_db.sh
/ngfw/usr/local/sf/bin/db_manage.sh
/ngfw/etc/certs/SRU_rel.crt
/ngfw/etc/certs/rel.crt
/ngfw/var/log/sf/verify_file_integ.log
/ngfw/var/tmp/merged-db/master.db
/ngfw/etc/certs/rewhich db_manage.sh
/tmp/all-process-smaps.txt
# Create a hash of the tar file: root@firepower:~# sha512sum SR-1234567890.tar 8833409f7affbae6ab19cdd8ca11153252f1176e147b3d4c373a7195e261da5ca72a3260a5aa1a5f991b12c1f6ac070d4ece31765e413f0e5e4afd6c95cf6ea5
SR-1234567890.tar root@firepower:~# # Copy the tar file off the platform using FTP or SCP: root@firepower:~# ftp 10.10.10.1 Connected to 10.10.10.1. 220 Microsoft FTP Service Name (10.10.10.1:admin): anonymous 331 Anonymous access allowed, send identity (e-mail name) as password. Password: 230 User logged in. Remote system type is Windows_NT. ftp> bin 200 Type set to I. ftp> put SR-1234567890.tar local: SR-1234567890.tar remote: SR-1234567890.tar 229 Entering Extended Passive Mode (|||59418|) 125 Data connection already open; Transfer starting. 100% |***********************************| 1990 KiB 104.76 MiB/s 00:00 ETA 226 Transfer complete. 2037760 bytes sent in 00:00 (100.48 MiB/s) ftp> quit 221 Goodbye.

Submit all command output and the files gathered in this step to the relevant TAC SR and proceed to the next section of this document.


Step Four – Verify Digitally Signed Image Authenticity

Cisco FXOS Software implements digitally signed system images on most platforms. Digitally signed Cisco FXOS Software uses asymmetric (public-key) cryptography which increases the security posture of Cisco Firepower devices by ensuring that the system image has not been altered.

Certain platforms running FXOS Software, such as the Cisco Firepower 2100 series of platforms, also support Cisco Secure Boot technologies. Cisco Secure Boot is a secure startup process that a Cisco device performs each time it boots up. Beginning with the initial power-on, a special purpose hardware device, known as the Trust Anchor module, verifies the integrity of the ROMMON code and the FXOS image via digital signatures as each are loaded. If any failures are detected, the user is notified of the error and the device will wait for the operator to correct the error. This prevents the network device from executing tainted network software.

For additional information see Trust Anchor Technology.

Note: The show software authenticity set of commands is only supported on Firepower platforms that incorporate Cisco Secure Boot technologies.

The authenticity and integrity of a system image file can be verified by using the following commands:

	
connect local-mgmt
cd bootflash:/
show file .boot_string
show software authenticity file <path/filename>
verify signature <path/filename>
	
	
An example of this procedure follows:

firepower# connect local-mgmt
firepower(local-mgmt)# cd bootflash:/
firepower(local-mgmt)# show file .boot_string
disk0:installables/switch/fxos-k8-fp2k-lfbff.2.4.1.216.SPA
firepower(local-mgmt)# show software authenticity file /installables/switch/fxos-k8-fp2k-lfbff.2.4.1.216.SPA      
File Name                     : <local>/fxos-k8-fp2k-lfbff.2.4.1.216.SPA
Image type                    : Release
    Signer Information
        Common Name           : abraxas
        Organization Unit     : FXOS
        Organization Name     : CiscoSystems
    Certificate Serial Number : 5BF5D36A
    Hash Algorithm            : SHA2 512
    Signature Algorithm       : 2048-bit RSA
    Key Version               : A

The Organization Unit, Organization Name, and Certificate Serial Number values can be viewed to verify whether the system image signature is valid.

Next, calculate a hash for the FXOS system image and verify the digital signature.


firepower(local-mgmt)# verify signature /installables/switch/fxos-k8-fp2k-lfbff.2.4.1.216.SPA          
Done!
Computed Hash   SHA2: 3688624cde65157037be90d66f9cd707
                      c781f6a1a9f6b84b175479628f73b1e3
                      6ef740867948a25ce9296598dab06009
                      6a1f5bf3cd3c32718d723bc09e269b3a

Embedded Hash   SHA2: 3688624cde65157037be90d66f9cd707
                      c781f6a1a9f6b84b175479628f73b1e3
                      6ef740867948a25ce9296598dab06009
                      6a1f5bf3cd3c32718d723bc09e269b3a                   
The digital signature of the file: fxos-k8-fp2k-lfbff.2.4.1.216.SPA verified successfully

It is also important to verify the authenticity and integrity of the running system image, and this can be accomplished with the following command:

show software authenticity running

An example of this procedure follows:

	
firepower(local-mgmt)# show software authenticity running
File Name                     : <local>/fxos-k8-fp2k-lfbff.2.4.1.216.SPA
Image type                    : Release
    Signer Information
        Common Name           : abraxas
        Organization Unit     : FXOS
        Organization Name     : CiscoSystems
    Certificate Serial Number : 5BF5D36A
    Hash Algorithm            : SHA2 512
    Signature Algorithm       : 2048-bit RSA
    Key Version               : A
	
	

The Organization Unit, Organization Name, and Certificate Serial Number values can be viewed to verify that the system image signature is valid, and the certificate serial number should be the same as the value obtained from the show software authenticity file command. In the examples above, the authenticity check of the FXOS Software image on bootflash: and the authenticity check of the running image both produce a value of 5BF5D36A.

Lastly, obtain a copy of the public keys with the following command:

show software authenticity keys


firepower(local-mgmt)# show software authenticity keys
Public Key #1 Information
--------------------------
Key Type              : Release (Primary)
Public Key Algorithm  : 2048-bit RSA
Modulus :
        B6:B9:BC:D3:C8:A1:BE:3A:B5:04:0F:21:6C:AA:AB:D6:
        CC:FE:7A:AD:CF:97:1B:57:FC:9A:1D:4B:5A:6D:D4:B0:
        7D:DB:77:FB:3F:A4:57:1A:08:4F:C1:6E:3F:CB:BF:E0:
        3C:99:9C:EE:F5:DD:3C:FC:C6:8D:98:49:29:00:B9:9B:
        DF:22:7E:73:83:FB:B5:78:68:4E:48:1A:5B:EE:83:81:
        B6:3B:2E:35:5B:C2:D0:B8:46:D6:45:13:23:21:44:DA:
        36:55:F9:09:5B:B1:88:8B:9A:28:0B:DA:44:DE:D2:F8:
        8B:17:CF:99:64:BE:2F:80:EF:13:6B:BC:A4:3E:DE:99:
        33:EF:E8:30:56:4C:DA:D5:D3:89:55:CC:BF:A2:22:1A:
        B7:64:FD:14:3A:7D:4F:00:DC:86:B5:35:18:C3:F3:FC:
        93:D4:BF:5E:FD:85:8C:28:4B:96:0F:B1:6D:1E:96:E7:
        05:1C:39:B7:1F:C7:F9:52:47:60:9C:96:FB:00:E2:2D:
        D9:08:2E:3A:87:0C:4F:3E:39:77:C7:FE:AC:D7:2D:23:
        AA:63:EB:2A:4D:13:98:C7:6A:B4:06:F9:1E:2D:B6:F8:
        10:80:EA:F4:E3:BF:C4:49:63:D0:5D:93:9F:96:54:76:
        BF:4D:83:7B:9D:CD:72:61:CC:EC:47:EA:91:EF:34:0B
Exponent              : 65537
Key Version           : A
Product Name          : FXOS
Public Key #2 Information
--------------------------
Key Type              : Release (Backup)
Public Key Algorithm  : 2048-bit RSA
Modulus :
        B6:B9:BC:D3:C8:A1:BE:3A:B5:04:0F:21:6C:AA:AB:D6:
        CC:FE:7A:AD:CF:97:1B:57:FC:9A:1D:4B:5A:6D:D4:B0:
        7D:DB:77:FB:3F:A4:57:1A:08:4F:C1:6E:3F:CB:BF:E0:
        3C:99:9C:EE:F5:DD:3C:FC:C6:8D:98:49:29:00:B9:9B:
        DF:22:7E:73:83:FB:B5:78:68:4E:48:1A:5B:EE:83:81:
        B6:3B:2E:35:5B:C2:D0:B8:46:D6:45:13:23:21:44:DA:
        36:55:F9:09:5B:B1:88:8B:9A:28:0B:DA:44:DE:D2:F8:
        8B:17:CF:99:64:BE:2F:80:EF:13:6B:BC:A4:3E:DE:99:
        33:EF:E8:30:56:4C:DA:D5:D3:89:55:CC:BF:A2:22:1A:
        B7:64:FD:14:3A:7D:4F:00:DC:86:B5:35:18:C3:F3:FC:
        93:D4:BF:5E:FD:85:8C:28:4B:96:0F:B1:6D:1E:96:E7:
        05:1C:39:B7:1F:C7:F9:52:47:60:9C:96:FB:00:E2:2D:
        D9:08:2E:3A:87:0C:4F:3E:39:77:C7:FE:AC:D7:2D:23:
        AA:63:EB:2A:4D:13:98:C7:6A:B4:06:F9:1E:2D:B6:F8:
        10:80:EA:F4:E3:BF:C4:49:63:D0:5D:93:9F:96:54:76:
        BF:4D:83:7B:9D:CD:72:61:CC:EC:47:EA:91:EF:34:0B
Exponent              : 65537
Key Version           : A
Product Name          : FXOS
Public Key #3 Information
--------------------------
Key Type              : Release (FEATURE KEY STORAGE)
Public Key Algorithm  : 2048-bit RSA
Modulus :
        C3:9E:B2:42:93:F2:F5:8A:E7:BA:8A:20:13:23:4A:24:
        39:93:C1:9E:83:32:D5:C7:87:38:54:14:1F:BC:66:8A:
        1A:F5:BA:B5:44:6A:5A:D0:B8:22:B2:3D:66:3D:34:A4:
        13:DF:3C:EB:02:34:97:D3:59:37:BE:86:D1:5C:40:F8:
        4B:F8:C0:7C:C8:92:0E:8F:C0:9B:49:88:8E:EE:31:B4:
        86:4A:3B:D6:D9:34:9F:CB:16:5F:1C:84:47:5A:9C:07:
        9A:12:F3:33:A2:EE:EB:76:8D:B3:C5:29:D2:D3:C4:ED:
        47:7C:70:E0:D3:80:00:36:C5:C1:BC:B0:45:EF:78:D5:
        62:02:5C:B4:35:0F:E9:D9:AD:5F:FF:F9:92:69:0C:01:
        5C:19:7F:E2:FE:0F:6B:8F:58:71:DB:E1:D7:F8:43:2F:
        AF:C1:80:F9:84:D0:AD:CA:A3:EC:C8:C4:C7:BE:48:53:
        EA:D5:31:44:63:B2:F8:3D:F4:C4:66:93:76:83:20:C0:
        1C:F4:B9:9A:3B:8A:FB:8A:D6:EC:9E:D8:35:B1:E1:F0:
        48:16:4C:49:16:65:05:60:8E:77:B4:AA:7A:E9:3F:E7:
        11:89:3E:98:4A:97:82:6E:09:18:4C:7C:8F:5B:45:89:
        78:16:C2:37:8F:3E:40:AE:35:09:D2:91:E6:7F:3C:FB
Exponent              : 65537
Key Version           : A
Product Name          : FXOS-CID

Submit all command output and any system images collected in this section to the relevant TAC SR and proceed to the next section of this document.


Step Five - Verify FTD Memory .text Segment Integrity

Execute the following commands from the Cisco FTD CLI prompt:

  
  system support diagnostic-cli
  enable
  
  

Then calculate a hash value for the .text memory segment and retrieve a copy of it by executing the following commands:


  verify /sha-512 system:memory/text
  copy system:memory/text ftp

An example of this procedure follows:

	
> system support diagnostic-cli
Attaching to Diagnostic CLI ... Press 'Ctrl+a then d' to detach.
Type help or '?' for a list of available commands.
firepower> enable
Password: 
firepower# verify /sha-512 system:memory/text !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[output truncated]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!Done! verify /SHA-512 (system:memory/text) = a03a15444f0995f578e9aa6cbc8feed2a3f2dd8ac8cca919b7b2b54836ba3d4b763372f58029e66fa64aafa8eea2b79d5f0c7ea65
cde0d813aef17e436e49b85 firepower# copy system:memory/text ftp Source filename [memory/text]? Address or name of remote host []? 10.10.10.1 Destination filename [text]? system.memory.text.bin !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! INFO: No digital signature found 71921664 bytes copied in 2.60 secs (35960832 bytes/sec)

It is highly recommended that a hash value be calculated on the copied memory segment file and compared to the hash value obtained on the recipient platform to ensure no errors were introduced during the file transfer process.

The example below utilizes the sha512sum utility which is included with most Linux distributions.

		
root@ftp-server:~# sha512sum system.memory.text.bin
a03a15444f0995f578e9aa6cbc8feed2a3f2dd8ac8cca919b7b2b54836ba3d4b763372f58029e66fa64aafa8eea2b79d5f0c7ea65cde0d813aef17e436e49b85
system.memory.text.bin root@ftp-server:~#

Note that the FTD verify command and the sha512sum utility both produce a SHA-512 hash value of a03a15444f0995f578e9aa6cbc8feed2a3f2dd8ac8cca919b7b2b54836ba3d4b763372f58029e66fa 64aafa8eea2b79d5f0c7ea65cde0d813aef17e436e49b85 for the system.memory.text.bin file.

Submit all command output (including all computed hash values) and any system images collected in this section to the relevant TAC SR, and proceed to the next section of this document.


Step Six – Cisco Firepower Crashinfo File/Core File

WARNING: Executing the tasks in this section will trigger a reload of the FXOS platform. Cisco recommends performing this task during a maintenance window. Cisco does not recommend performing this task if additional forensic information needs to be collected because a reload of the device may cause the loss of information vital to a forensic investigation. Please ensure you have a copy of the original device configuration and the appropriate authorization to initiate a reload of the platform in question before proceeding.

This procedure outlines how to obtain a crashinfo file and a core dump from a Cisco FXOS device.

The crashinfo file is saved in the root of the Cisco FXOS file system by default and the core dump may be placed in the underlying FTD file system or the coredumpfsys filesystem dependent on the version of software running on the system. Storage space required may vary from several hundred megabytes to several gigabytes in size depending on device model. Be sure there is enough space on the destination flash or disk device to accommodate the crashinfo file and core dump file.

To initiate the crashinfo dump process, execute the following commands.

An example of this procedure follows:


> system support diagnostic-cli
Attaching to Diagnostic CLI ... Press 'Ctrl+a then d' to detach.
Type help or '?' for a list of available commands.
firepower> enable
Password: 
firepower# crashinfo force page-fault
WARNING: This command will force a crash and cause a
         reboot. Do you wish to proceed? [confirm]: 
:Saved Crash
Process Name: lina
Signal No.: 11
Thread id: 1363
Register dump from crashing thread
R00: 0x0000000000000000
R01: 0x0000000000000001
R02: 0x0000002009499f80
R03: 0x0000000000000059
R04: 0x000000000000000a
R05: 0xfffffffffffffffd
R06: 0x0000000000000000
R07: 0x0000000000000000
R08: 0x000000ffd12f2da8
R09: 0x000000ffd12fa3a0
[output truncated]

Show tech-support output is captured and saved.
Crashinfo file created: /mnt/disk0/crashinfo_lina.1359.20200217.163743
Rebooting... (status 0x8b)

When the crashinfo process is complete, the Firepower platform will reboot.

Once the platform has rebooted, connect to the FTD CLI, enter expert mode, and copy the core file to disk0 so that it can be copied off the platform by executing the following commands:


expert
sudo su – 
cd /var/data/cores
cp  /mnt/disk0/.

Note: the sudo su - command must be executed after entering expert mode to ensure the correct privileges are obtained to copy the core file from one disk partition to another.

An example of this procedure follows:


> expert
admin@firepower:~$ sudo su -
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.
Password: 

root@firepower:common# cd /var/data/cores
root@firepower:cores# ls -la
total 104944
drwxrwxrwx 4 root root      4096 Feb 17 16:37 .
drwxr-xr-x 3 root root        60 Feb 17 16:40 ..
-rw-r--r-- 1 root root 107325707 Feb 17 16:38 core.lina.core-2.11.1359.1581957475.gz
drwx------ 2 root root     16384 May  8  2019 lost+found
drwxr-xr-x 3 root root      4096 May  8  2019 sysdebug
root@firepower:cores# cp core.lina.core-2.11.1359.1581957475.gz /mnt/disk0/.
root@firepower:cores# exit
logout
admin@firepower:~$ exit
logout
>

Next, enter the diagnostic CLI and calculate hash values for the crashinfo and core files by issuing the following commands:

	
system support diagnostic-cli
verify /sha-512 <filename>
	
	

An example of this procedure follows:

	
> system support diagnostic-cli
Attaching to Diagnostic CLI ... Press 'Ctrl+a then d' to detach.
Type help or '?' for a list of available commands.
firepower> enable
Password: 
firepower# dir
Directory of disk0:/

409523893  drwx  4096         22:22:00 Nov 21 2019  .private
270331987  drwx  49           19:50:22 May 08 2019  log
402739781  drw-  25           16:37:55 Feb 17 2020  coredumpinfo
270169092  -rwx  5175         15:20:29 Nov 21 2019  backup-config.cfg
270169093  -rwx  5175         15:20:30 Nov 21 2019  startup-config
270169106  -rwx  4768         15:20:32 Nov 21 2019  modified-config.cfg
270185408  -rw-  0            22:22:05 Nov 21 2019  hitcnt_del_ruleid_list
270038749  -rw-  113421       16:37:55 Feb 17 2020  crashinfo_lina.1359.20200217.163743
270044144  -rw-  107325707    18:14:24 Feb 17 2020  core.lina.core-2.11.1359.1581957475.gz
6 file(s) total size: 107454246 bytes
53751054336 bytes total (38843449344 bytes free/72% free)

firepower# verify /sha-512 disk0:/crashinfo_lina.1359.20200217.163743
!!!!!!!!!!!!!!!!!!!!!!!!!!!!Done!
verify /SHA-512 (disk0:/crashinfo_lina.1359.20200217.163743) =
639a8e8289a876532b98cffd7ce719d163eb4fe24374250249f56bb1187c4fccafba

70f6cfe2f5db7cba4b89e0a8b7bf6bcc193a9d7a6004bf049fe56d032294
firepower# verify /sha-512 disk0:/core.lina.core-2.11.1359.1581957475.gz !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[output truncated]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! verify /SHA-512 (disk0:/core.lina.core-2.11.1359.1581957475.gz) =
60afbbe6159c0941f5c6918d312c2c34b46c0c21936d9a9e661e5fb20d95161e9
07839a9b101be774282212e0d39bc76cac5b63d100f7d4d80af543bc557025d
firepower# exit firepower> exit Console connection detached. >

It is highly recommended that hash values be calculated on the crashinfo and core files obtained in this section so that any errors introduced by subsequent copying or transmission can be reliably detected.

The last step is to copy the crashinfo and core files to a secure location. The following example transfers the files to an FTP server using the file copy command:

	
> file copy 10.10.10.1 anonymous / crashinfo_lina.1359.20200217.163743
Enter password for anonymous@10.10.1.1: 
Copying crashinfo_lina.1359.20200217.163743
Copy successful.
> file copy 10.10.10.1 anonymous / core.lina.core-2.11.1359.1581957475.gz
Enter password for anonymous@10.10.1.1: 
Copying core.lina.core-2.11.1359.1581957475.gz
Copy successful.
	
	

Submit all command output, hash values, crashinfo and core files collected in this section to the relevant TAC SR, and proceed to the next section of this document.


Step Seven – ROMMON Settings Check

The ROM Monitor firmware of the Firepower platform is executed when the appliance is powered up or reset. The firmware initializes the platform hardware and boots the FXOS operating system software. Because the ROM Monitor settings are persistent if they have been synced to NVRAM, information about the ROM Monitor variable values could indicate an attempt to influence the FXOS boot sequence. The set command can be used while in the ROM Monitor prompt to see the value of the ROM Monitor variables.

ROM Monitor mode is accessed by rebooting the Firepower appliance and pressing the BREAK or ESC key during the reload process when prompted as depicted in the example below.


firepower# connect local-mgmt 
firepower(local-mgmt)# reboot
Before rebooting, please take a configuration backup.
Do you still want to reboot? (yes/no):yes

Broadcast message from admin@firepower (Fri Aug 30 14:17:09 2019):
All shells being terminated due to system /sbin/reboot

Threat Defense System: CMD=-stop, CSP-ID=cisco-ftd.6.3.0.83__ftd_001_JMX2312Y09KWMZZP41, FLAG=''
Cisco FTD stopping ...
Stopping Cisco Firepower 2110 Threat Defense......ok
Skipping sfifd for this platform...
Stopping nscd...
Stopping nscd...                                                      [  OK  ]
Turning off swapfile /ngfw/Volume/.swaptwo
Stopping system log daemon...
Stopping system log daemon...                                         [  OK  ]
Stopping Threat Defense ...
Stopping Threat Defense ...                                           [  OK  ]
Cisco FTD stopped successfully.
Stopping all devices.

[output truncated]

Rebooting... [63349.896618] reboot: Restarting system
******************************************************************************
Cisco System ROMMON, Version 1.0.06, RELEASE SOFTWARE
Copyright (c) 1994-2017  by Cisco Systems, Inc.
Compiled Wed 11/01/2017 18:38:59.66 by builder
******************************************************************************
Current image running: Boot ROM0
Last reset cause: ResetRequest
DIMM_1/1 : Present
DIMM_2/1 : Absent
Platform FPR-2110 with 16384 MBytes of main memory
BIOS has been successfully locked !!
MAC Address: 00:fd:22:61:8b:f1

Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.
Boot interrupted.  

rommon 1 > 

The following example shows the output of the ROM Monitor set command on a Cisco Firepower platform:

	
rommon 1 > set
ADDRESS=
NETMASK=
GATEWAY=
SERVER=
IMAGE=
CONFIG=
PS1="rommon ! > "
	
	

The example above depicts a platform where the ROM Monitor values are at their default values and have not been altered.

To return the Firepower platform to normal operation, simply issue the boot command at the ROM Monitor prompt as depicted in the following example.

	
rommon 2 > boot
Located '.boot_string' @ cluster 837007.
#
Located 'installables/switch/fxos-k8-fp2k-lfbff.2.4.1.216.SPA' @ cluster 598122.
######################################################################################################################
###################################### [output truncated]

Submit all command output obtained in this section to the relevant TAC SR.


Related Documentation

Additional information about Cisco Software Integrity Assurance, as well as forensic investigation procedures for other platforms, can be found at the following link:

Cisco Security Tactical Resources

https://www.cisco.com/c/en/us/about/security-center/intelligence/asa-integrity-assurance.html


Cisco Firepower Device Forensic Response Checklist


Step 1 – Create the Firepower Device Problem Description

     Device Problem Description uploaded to SR

Step 2 – Document the FTD Runtime Environment

     Output of show tech-support uploaded to SR

     Output of dir all-filesystems uploaded to SR

     Output of other show commands uploaded to SR (Optional)

Step 3 – Verify FTD System File Integrity

     Output of find /ngfw/var/sf/.icdb/* and hashes uploaded to SR

     Output of which command executed on shell scripts uploaded to SR

     Shell scripts, certificates, log file, and hash database added to .tar file

     .tar file and its associated hash value uploaded to SR

Step 4 – FXOS Digitally Signed Image Authenticity Verification

     Output of show software authenticity file uploaded to SR

     Output of show software authenticity running uploaded to SR

     Output of show software authenticity keys uploaded to SR

Step 5 – Verify Memory .text Segment Integrity

     Output of verify on memory text segment uploaded to SR

     Copy of memory text segment uploaded to SR

Step 6 – FTD Crashinfo File/Core File

     Output of crashinfo uploaded to SR

     crashinfo file uploaded to SR

     Core file uploaded to SR

     Hash values of crashinfo and core files uploaded to SR

Step 7 – Firepower ROM Monitor Variable Check

     Output of set command uploaded to SR

Revision History

Version Date Author Comments
1.0 5/14/20 Dan Maunz, Jason Barnes  
       
       

 


This document is part of the Cisco Security portal. Cisco provides the official information contained on the Cisco Security portal in English only.

This document is provided on an “as is” basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information in the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document without notice at any time.


Back to Top