Guest

Cisco Security

Cisco ASA Forensic Investigation Procedures for First Responders


Introduction

Prerequisites

Step One - ASA Device Problem Description

Step Two - Document the ASA Runtime Environment

Step Three - ASA Image File Hash Verification

Step Four - ASA Core File/Memory Dump

Step Five - ROMMON Settings Check

Step Six - SSL VPN Configuration Integrity Check

Acknowledgments

Related Documentation

ASA Device Forensic Checklist

Revision History




Introduction

This document provides guidance for collecting forensic evidence from devices running Cisco ASA Software that are suspected of compromise or tampering. The document describes a number of commands that can be run to gather evidence for an investigation, along with the respective output that should be collected upon running these commands. This document also provides information on how to perform integrity checks on an ASA device’s system images, and includes a procedure for collecting a core file/memory dump from an ASA device.

Note: This document applies only to Cisco ASA Software and to no other Cisco operating systems. This document does not apply to any of the service modules that may be installed or running within a Cisco ASA device, such as an IPS or FirePOWER module.

Important: It is extremely important when triaging a network device for compromise or tampering that it is not rebooted. Rebooting a device during an initial assessment will irrecoverably lose all volatile information contained within the device. (e.g. RAM contents, ARP and routing tables, NAT translations, ACL hit and drop counts, etc.)

Note: It is highly recommended that a device suspected of tampering or compromise be isolated from the network prior to conducting an initial forensic examination. This may prevent remote unloading of any implants or malware installed on the device and will prevent an adversary from monitoring commands entered on the device under investigation.

If you require assistance, or have questions regarding the following procedures, contact the Product Security Incident Response Team (PSIRT).

This document contains seven main sections:

  1. ASA Device Problem Description – Describes why the platform is a candidate for forensic examination
  2. ASA Runtime Environment – Collects platform configuration and runtime state
  3. ASA Image File Verification – Examines system image hashes for inconsistencies
  4. Digitally Signed Image Verification – Examines system and running images for proper signing characteristics
  5. Core File/Memory Dump – Obtains a core dump of the running ASA image and contents of memory
  6. ROM Monitor Variables – Examines ROM monitor settings for remote system image loading
  7. SSL VPN Integrity Checks – Examines the webvpn configuration (if enabled) for tampering


Prerequisites

The procedures outlined in this document assume the reader has a basic understanding of Cisco ASA Software command syntax.

A valid cisco.com account is required to view individual ASA Software and ASA firmware file hashes for software file integrity checking. For customers without a cisco.com account, a publicly available comprehensive list of file hashes (Bulk Hash File) can be downloaded from: https://www.cisco.com/c/en/us/about/trust-center/downloads.html

A Cisco Technical Assistance Center (TAC) service request (SR) is required for the device in question as the procedures outlined in this document assume that the information gathered in each step will be uploaded to a TAC SR to be subsequently analyzed by Cisco engineers for indications of compromise or tampering.


Step One – ASA Device Problem Description

Describe in as much detail as possible WHY the device is a candidate for forensic examination. Are there configuration changes that cannot be explained? Is there unusual traffic originating from or terminating on the device? Are there anomalous entries in the device logs or in syslog messages? Is the device exhibiting odd behavior than cannot be attributed to a misconfiguration or a software / hardware defect?

Submit the problem description collected in this section to the relevant TAC SR, and proceed to the next section of this document.


Step Two – Document the ASA Runtime Environment

The initial stage of evidence gathering is completed by issuing a show tech-support command and a dir all-filesystems command. These commands must be executed in enable mode (i.e. privileged EXEC mode), and some of the output produced may vary depending on the particular ASA Software version and/or features supported or configured on the device.

Execute each of the following commands in enable mode and record the output:

show tech-support detail
dir all-filesystems

The following list of commands may also be executed to gather additional information relevant to the current operating state of the device. Although the output of these commands is not required to perform a forensic analysis of an ASA platform, it may provide additional information regarding any unauthorized changes made to the device if compromise is suspected.

Execution of the following commands is optional.

terminal length 0
show history
show clock detail
show startup-config
show reload
show process
show kernel process detail
show kernel ifconfig
show kernel module
show logging
show route
show eigrp nei
show ospf nei
show bgp summary
show arp
show ip address
show interface ip brief
show nat detail
show snmp-server user
show snmp-server group
show ipv6 interface brief
show ipv6 route
show conn all
show xlate
show aaa login-history

Submit all command output collected in this section to the relevant TAC SR, and proceed to the next section of this document.


Step Three – ASA Image File Hash Verification

Access the CLI of the device running Cisco ASA Software and issue the following command in enable mode:

show version

Note the location and filename of the ASA system image file and then execute the following command:

verify /sha-512 location:filename

Alternatively, an MD5 hash value can be calculated with the following command:

verify /md5 location:filename

An example of this procedure is as follows:


ciscoasa# show version
Cisco Adaptive Security Appliance Software Version 9.8(2)20 
Firepower Extensible Operating System Version 2.2(2.63)
Device Manager Version 7.9(1)151
Compiled on Fri 02-Feb-18 06:10 PST by builders
System image file is "disk0:/asa982-20-lfbff-k8.SPA"
Config file at boot was "startup-config"
[output truncated]

ciscoasa# verify /sha-512 disk0:/asa982-20-lfbff-k8.SPA
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[output truncated]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Done!
verify /SHA-512 (disk0:/asa982-20-lfbff-k8.SPA) = 24428310bf17
c94c58c73f7488111fcd92be4be9dce3e501c8db0f8c6a6ff934ae4ca540ea
49910ba6b1cdedcbada99db167dbb3eabd69ca27b88c4292e0be16

The computed hash calculated by the verify command should match the SHA-512 value listed on CCO or in the Bulk Hash File for that particular image file.

Repeat the previous procedure for any other system image file located on the file systems. A comprehensive list of all files can be viewed by executing the following command:

dir all-filesystems

If any of the image file hashes show inconsistencies, copy the image file in question to a secure location if possible.


copy : ftp: 
Address or name of remote host []? 
Destination filename []?         
                  

It is highly recommended that a hash value be calculated on the copied system image file and compared to the hash value obtained on the platform to ensure no errors were introduced during the file transfer process.

The following example utilizes the sha512sum utility, which is included with most Linux distributions.


root@ftp-server:~# sha512sum asa982-20-lfbff-k8.SPA
24428310bf17c94c58c73f7488111fcd92be4be9dce3e501c8db0f8c6a6ff934ae4ca540ea49910ba6b1cdedcbada99db167dbb3eabd69ca2
7b88c4292e0be16asa982-20-lfbff-k8.SPA
root@ftp-server:~#

Note that the ASA verify command and the sha512sum utility both produce an SHA-512 hash value of 24428310bf17c94c58c73f7488111fcd92be4be9dce3e501c8db0f8c6a6ff934ae4ca540ea49910ba 6b1cdedcbada99db167dbb3eabd69ca27b88c4292e0be16 for the asa982-20-lfbff-k8.SPA file.

Submit all command output (including all computed hash values) and any system images collected in this section to the relevant TAC SR, and proceed to the next section of this document.


Step Four – ASA Core File/Memory Dump

CAUTION: This section contains commands that alter the ASA device configuration. Please ensure you have a copy of the original device configuration and the appropriate authorization to make changes to the platform in question prior to proceeding with this procedure.

WARNING: Executing the tasks in this section will trigger an immediate reload of the ASA platform. Cisco recommends performing this task during a maintenance window. Cisco does not recommend performing this task if additional forensic evidence needs to be collected, as a reload of the device may cause the loss of information vital to a forensic investigation.

This procedure describes how to configure a Cisco ASA device to obtain a dump of platform memory. The core dump is saved on the Cisco ASA file system in the /coredumpfsys directory and the storage space required may vary from several hundred megabytes to several gigabytes in size depending on the device model. Be sure there is enough space on the destination ASA flash or disk file system to accommodate the coredump.

To configure the system to generate a coredump, use the coredump enable command in configuration mode:


conf t
coredump enable filesystem filesystem

An example of this procedure follows:


ciscoasa# conf t
ciscoasa(config)# coredump enable filesystem disk0: 

WARNING: Enabling coredump on an ASA5512 platform will delay the reload of
the system by up to 30 minutes in the event of software forced
reload. The exact time depends on the size of the coredump generated.
Proceed with coredump filesystem allocation of 1000 MB on 'disk0:' (Note this may take a while) ? [confirm] filesys_image created ok: disk0:coredumpfsysimage.bin Making coredump file system image!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!! Coredump file system image created & mounted successfully /dev/loop0 on /mnt/disk0/coredumpfsys type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso885 9-1,shortname=mixed,errors=remount-ro)

To initiate the core dump process, execute the following command in enable mode:

crashinfo force page-fault

The core dump may take some time to complete, depending on the amount of physical memory (RAM) installed on the device.


ciscoasa# crashinfo force page-fault 
WARNING:  This command will force a crash and cause a
          reboot. Do you wish to proceed? [confirm]: 
Thread Name: ci/console
Page fault: Address not mapped
        r8 0x00007f00fb30a9c8
        r9 0x0000000000000001
       r10 0x00000000e150e560
       r11 0x0000000000000000
       r12 0x0000000000000005

[output truncated]

Begin to dump crashinfo to flash....
End of console dump.
Do 'show crashinfo' after reboot to retrieve other crash information
Process shutdown finished
[1201369.566294] 
[1201369.566294] Writing coredump file to flash. Please do not reload.
[1201369.566294] 
[1201369.566294] Coredump starting....
[1201369.778010] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Coredump completed Rebooting... (status 0x8b)

When the core dump process is complete, the ASA will reboot.

The core dump is written to a compressed file located on the /coredumpfs file system. The name of the file can be displayed using the following command:

dir disk0:/coredumpfsys

ciscoasa# dir disk0:/coredumpfsys 

Directory of disk0:/coredumpfsys/
122  -rwx  74695443 07:00:18 Jun 13 2018 core_smp.2018Jun13_065957.1770.11.gz
4118732800 bytes total (2911567872 bytes free)

It is highly recommended that hash values be calculated on the core files obtained in this section so that any errors introduced by subsequent copying or transmission can be reliably detected.

Submit all command output and core files collected in this section to the relevant TAC SR, and proceed to the next section of this document.

System Memory Dump (Alternate Procedure)

Obtaining a core dump as outlined in the previous procedure is the preferred method for retrieving a complete copy of the device memory image. However, in cases where a device cannot be rebooted, the following commands can be used to retrieve the most important memory segment.

Note: The ASA platform must be running software version 9.6.2 or higher in order to successfully execute this procedure.


show version
verify /sha-512 system:memory/text
copy system:memory/text ftp

An example of this procedure follows:


ciscoasa# show version 
Cisco Adaptive Security Appliance Software Version 9.8(2)20 
Firepower Extensible Operating System Version 2.2(2.63)
Device Manager Version 7.9(1)151
Compiled on Fri 02-Feb-18 06:10 PST by builders
System image file is "disk0:/asa982-20-lfbff-k8.SPA"
Config file at boot was "startup-config"
[output truncated]

ciscoasa# verify /sha-512 system:memory/text
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [output truncated]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Done!
verify /SHA-512 (system:memory/text) = ecebb9f65a04fbeb985456dafea5866af992a9045458e9e89212c4a06f84058846e9c7318c75
80be61d0af5819434f07e1c21aa527986c1ac8bc0d13a4ae6fc1 ciscoasa# copy system:memory/text ftp Source filename [memory/text]? Address or name of remote host []? 10.10.10.1 Destination filename [text]? system.memory.text.bin !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! INFO: No digital signature found 70193152 bytes copied in 4.70 secs (17548288 bytes/sec)

It is highly recommended that a hash value be calculated on the copied memory segment file and compared to the hash value obtained on the platform to ensure no errors were introduced during the file transfer process.

The following example utilizes the sha512sum utility, which is included with most Linux distributions.


root@ftp-server:~# sha512sum system.memory.text.bin
ecebb9f65a04fbeb985456dafea5866af992a9045458e9e89212c4a06f84058846e9c7318c75
80be61d0af5819434f07e1c21aa527986c1ac8bc0d13a4ae6fc1
system.memory.text.bin root@ftp-server:~#

Note that the ASA verify command and the sha512sum utility both produce an SHA-512 hash value of ecebb9f65a04fbeb985456dafea5866af992a9045458e9e89212c4a06f84058846e9c7318c7580be 61d0af5819434f07e1c21aa527986c1ac8bc0d13a4ae6fc1 for the system.memory.text.bin file.

Submit all command output (including all computed hash values) and any system images collected in this section to the relevant TAC SR, and proceed to the next section of this document.


Step Five – ROMMON Settings Check

The ROM monitor firmware of the ASA platform is executed when the ASA is powered up or reset. The firmware initializes the platform hardware and boots the ASA operating system software. Because the ROM monitor settings are persistent if they have been synced to NVRAM, information about the ROM monitor variable values could indicate an attempt to influence the Cisco ASA boot sequence. The set command can be used while in the ROM monitor prompt to see the value of the ROM monitor variables.

ROM monitor mode is accessed by rebooting the ASA device and pressing the BREAK or ESC key during the reload process when prompted as depicted in the following example:


ciscoasa# reload
Proceed with reload? [confirm] 
ciscoasa# 
***
*** --- START GRACEFUL SHUTDOWN ---
Shutting down isakmp
Shutting down webvpn
Shutting down sw-module
Shutting down License Controller
Shutting down File system
***
*** --- SHUTDOWN NOW ---
Process shutdown finished
Rebooting... (status 0x9)

[output truncated]

Booting from ROMMON
Cisco Systems ROMMON Version (2.1(9)8) #1: Wed Oct 26 17:14:40 PDT 2011
Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.
Boot interrupted.
Management0/0
Link is DOWN
MAC Address: 0006.f62b.8587
Use ? for help.
rommon #0>

The following example shows the output of the ROM monitor set command on a Cisco ASA platform:


rommon #0> set
ROMMON Variable Settings:
  ADDRESS=0.0.0.0
  SERVER=0.0.0.0
  GATEWAY=0.0.0.0
  PORT=Management0/0
  VLAN=untagged
  IMAGE=
  CONFIG=
  LINKTIMEOUT=20
  PKTTIMEOUT=4
  RETRY=20

The previous example depicts a platform where the ROM monitor values are at their default values and have not been altered.

To return the ASA platform to normal operation, issue the boot command at the ROM monitor prompt as depicted in the following example:


rommon #1> boot
Launching BootLoader...
Boot configuration file contains 1 entry.

Loading disk0:/asa982-smp-k8.bin... Booting...
Platform ASA5512

[output truncated]

Submit all command output obtained in this section to the relevant TAC SR, and proceed to the next section of this document.


Step Six – SSL VPN Configuration Integrity Check

The Cisco ASA platform supports the termination of clientless SSL VPN connections.When this feature is configured, the Cisco ASA will enable an internal web server that listens for remote connections to the SSL VPN portal.

The SSL VPN portal configuration is stored in XML files and additional functionality can be added through the importation of client plug-ins. These ActiveX or Java plug-ins enable features for SSL VPN sessions such as Remote Desktop Protocol (RDP), Secure Shell (SSH) and Telnet, and Virtual Network Computing (VNC) to name a few.

The XML configuration files and imported client plug-ins can both be sources of web-based exploit code, such as java code injection and malicious iframes, and should be examined if these features are enabled and the ASA platform is suspected of being compromised and/or tampered with.

Verifying that SSL VPN portal is enabled and retrieving the XML configuration code can be accomplished with the following commands:


show run | begin webvpn
export webvpn customization dfltCustomization filesystem:/filename

An example of this procedure follows:


ciscoasa# show run | begin webvpn
webvpn
 enable outside
 anyconnect image disk0:/anyconnect-win-4.5.04029-webdeploy-k9.pkg 1
 anyconnect image disk0:/anyconnect-macos-4.5.04029-webdeploy-k9.pkg 2
 anyconnect image disk0:/anyconnect-linux64-4.5.04029-webdeploy-k9.pkg 3
 anyconnect enable
 tunnel-group-list enable
[output truncated]

ciscoasa# export webvpn customization dfltCustomization disk0:/webvpn.txt
%INFO: Customization object 'DfltCustomization' was exported to disk0:/webvpn.txt

Note: The presence of webvpn in the configuration does not by itself indicate that SSL VPN is enabled, it must also be accompanied by an enable <interface_name> statement.

In the previous example, an SSL VPN portal is configured on the outside interface (as indicated by the “enable outside” statement), and the DfltCustomization configuration file is copied to ‘webvpn.txt’ on disk0, which may now be copied off the ASA platform via FTP for analysis.

Checking the Integrity of SSL VPN Plugins

Any imported SSL VPN plug-ins can be enumerated and copies of the plug-ins extracted from the ASA using the following commands:


show import webvpn plug-in detail
export webvpn plug-in protocol protocol url/filename

Note: If no plug-ins have been imported, the show import webvpn plug-in detail command will not produce any output.

An example of this procedure follows:


ciscoasa# show import webvpn plug-in detail
ssh,telnet   pHjJoOO34pptovI2yD594rvZKc4= Sun, 22 Dec 2013 02:07:29 GMT

ciscoasa# export webvpn plug-in protocol ssh,telnet ftp://10.10.10.1/ 
ssh.12.21.2013.jar

The first command enumerates all of the SSL VPN plug-ins that have been imported into the ASA platform, and the second command exports specific plug-ins to a FTP server where hash values can be calculated and compared with values listed on CCO or in the Bulk Hash File for that particular plug-in file.

Note: The string pHjJoOO34pptovI2yD594rvZKc4= in the example above is a base-64 encoded SHA-1 hash of the ssh,telnet plug-in file. It is recommended that plug-ins be copied off the device and a SHA-512 or MD5 hash calculated as only SHA-512 and MD5 hash values are listed on the CCO website.

The following example utilizes the sha512sum utility, which is included with most Linux distributions.


root@ftp-server:~# sha512sum ssh.12.21.2013.jar
9d38ef2071cd18eadd875648323bb8ee5b9e0b5d12a2191400d6ab7168a58e325a860de787bfca487a1abaf4ed58e8efc38ef45de9ef5e
5534adcddfde9eb95f
ssh.12.21.2013.jar root@ftp-server:~#

The following example utilizes the md5sum utility, which is included with most Linux distributions.


root@ftp-server:~# md5sum ssh.12.21.2013.jar      
a3d64b3d0291bb992e43ab43235dd9cc  ssh.12.21.2013.jar
root@ftp-server:~#

Submit all command output obtained in this section to the relevant TAC SR.


Acknowledgments

The author would like to thank all members of the Customer Experience Security Programs (CXSP) and Advanced Security Initiatives Group (ASIG) who provided their expertise and support during the writing of this document.


ASA Device Forensic Response Checklist

Step 1 - Create the ASA Device Problem Description

    Device Problem Description uploaded to SR 

Step 2 - Document ASA Runtime Environment

    Output of show tech-support uploaded to SR 

    Output of dir all-filesystems uploaded to SR 

    Output of other show commands uploaded to SR (Optional) 

Step 3 - ASA Image File Hash Verification

    Output of verify on system image files uploaded to SR

    Image files with hash inconsistencies uploaded to SR

Step 4 - ASA Digitally Signed Image Authenticity Verification

    Output of show software authenticity file uploaded to SR 

    Output of show software authenticity running uploaded to SR 

Step 5 - ASA Core File/Memory Dump

    Output of crashinfo uploaded to SR 

    Core file uploaded to SR 

    Output of verify on memory text segment uploaded to SR (Alternate) 

    Copy of memory text segment uploaded to SR (Alternate) 

Step 6 - ASA ROM Monitor Variable Check

    Output of set command uploaded to SR 

Step 7 - SSL VPN Configuration Integrity Check (If applicable)

    Output of export webvpn command uploaded to SR 

    Output of show import webvpn command uploaded to SR 

    Plug-in files with hash inconsistencies uploaded to SR 


Revision History

Version Date Author Comments
1.0 8/19/2019 Dan Maunz Initial public release.
       
       

 


This document is part of the Cisco Security portal. Cisco provides the official information contained on the Cisco Security portal in English only.

This document is provided on an “as is” basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information in the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document without notice at any time.


Back to Top