Guest

Cisco Security

Cisco IOS Software Forensic Investigation Procedures for First Responders


Introduction

Prerequisites

Step One - Cisco IOS Device Problem Description

Step Two - Document the IOS Runtime Environment

Step Three - Cisco IOS Image File Hash Verification

Step Four - ROMMON Upgrade Check

Step Five - Cisco IOS Core File / Memory Dump

Step Six - Analysis With the verify Command

Acknowledgments

Related Documentation

Cisco IOS Device Forensic Response Checklist

Revision History




Introduction

This document provides guidance for collecting evidence from Cisco IOS devices that are suspected of having been compromised or tampered with. It outlines a number of commands that can be run to gather evidence for an investigation, along with the output that should be collected after running these commands. This document also provides information on how to perform integrity checks on a device’s IOS images, a check to see if a ROMMON upgrade has been applied, and includes a procedure for collecting a core file/memory dump from a Cisco IOS device.

Note: It is extrememly important when triaging a network device that it is not rebooted. Rebooting a device during an initial assessment will irrecoverably lose all volatile information contained within the device. (e.g. RAM contents, arp & routing tables, NAT translations, ACL hit & drop counts, etc.)

Note: It is highly recommended that a device suspected of tampering or compromise be isolated from the network prior to conducting an initial forensic examination. This may prevent remote unloading of any implants or malware installed on the device and will prevent an adversary from monitoring commands entered on the device under investigation.

If you require assistance or have questions regarding the following procedures, contact the Product Security Incident Response Team (PSIRT)

The body of this document contains six sections:

1.      Cisco IOS Device Problem Description – Describes why the platform is a candidate for forensic examination.

2.      Cisco IOS Runtime Environment – Collects platform configuration and runtime state.

3.      Cisco IOS Image File Verification – Examines system image hashes for inconsistencies.

4.      ROMMON Upgrade Check – Examines the ROM monitor region for an upgraded image.

5.      Core File/Memory Dump – Obtains a core dump of the running IOS image and contents of memory.

6.      Analysis with the verify command – Provides an alternate method of analysis if a core dump cannot be performed.

 

Prerequisites

The procedures outlined in this document assume that the reader has a basic understanding of Cisco IOS Software command syntax.

A valid cisco.com account is required to view individual IOS and ROMMON file hashes for software file integrity checking. A publicly available comprehensive list of file hashes (Bulk Hash File) can be downloaded from: https://www.cisco.com/c/en/us/about/trust-center/downloads.html

A Cisco Technical Assistance Center (TAC) service request (SR) is required for the device in question as the procedures outlined in this document assume that the information gathered in each step will be uploaded to a TAC SR to be analyzed by Cisco engineers for indications of compromise or tampering.


Step One – Cisco IOS Device Problem Description

Describe in as much detail as possible WHY the device is a candidate for forensic examination. Are there configuration changes that cannot be explained? Is there unusual traffic originating from or terminating on the device? Are there anomalous entries in the device logs or in syslog messages? Is the device exhibiting odd behavior that cannot be attributed to a misconfiguration or a software/hardware defect?

Submit the problem description collected in this section to the relevant TAC SR and proceed to the next section of this document.


Step Two – Document the Cisco IOS Runtime Environment

The initial stage of evidence gathering is completed by issuing a show tech-support command and a dir all-filesystems command. These commands must be executed in enable mode (privileged EXEC mode), and some of the output produced may vary depending on the particular IOS version and/or configured features.

Execute each of the following commands in enable mode and record the output:


show tech-support
dir all-filesystems

Note: This section of Step 1 is optional. The following list of commands may also be executed to gather additional information relevant to the current operating state of the device. Although the output of these commands is not required to perform a forensic analysis of an IOS platform, they may provide additional information regarding any unauthorized changes made to the device if compromise is suspected.


terminal length 0
show history all
show clock detail
show startup-config
show reload
show ip route
show ip eigrp nei
show ip ospf nei
show ip bgp summary
show cdp nei detail
show ip arp
show ip interface
show ip interface brief
show tcp brief all
show sockets
show ip nat translations verbose
show ip cache flow
show ip cef
show snmp user
show snmp group
show snmp community
show ipv6 interface brief
show ipv6 route
show logging
show processes

Alternatively, the following Tcl script can be copied and pasted into the command line to collect the output from the commands listed above.

Note that the script may not be fully functional on all versions of IOS Software.


tclsh
##
## Cisco Router Triage Script
## v1.0
##
## IOS/ROMMON Configuration & User Info
##
exec "terminal length 0"
exec "show history all"
exec "show clock detail"
exec "show startup-config"
exec "show reload"
##
## Network, SNMP, and ACL Info
##
exec "show ip route"
exec "show ip eigrp nei"
exec "show ip ospf nei"
exec "show ip bgp summary"
exec "show cdp nei detail"
exec "show ip arp"
exec "show ip interface"
exec "show ip interface brief"
exec "show tcp brief all"
exec "show sockets"
exec "show ip nat translations verbose"
exec "show ip cache flow"
exec "show ip cef"
exec "show snmp user"
exec "show snmp group"
exec "show snmp community
exec "show ipv6 interface brief"
exec "show ipv6 route"
##
## Local Logging, Process and Memory Info
##
exec "show logging"
exec "show processes"
##
##
##
tclquit

Submit all command or script output collected in this section to the relevant TAC SR and proceed to the next section of this document.


Step Three – Cisco IOS Image File Hash Verification

Access the command line of the Cisco IOS device and issue the following command in enable mode:

show version

Note the location and filename of the system image file and then execute the following command:

verify location:filename

An example of this procedure follows:


Router1#show version
Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M12a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 04-Oct-16 03:37 by prod_rel_team
ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

Router1 uptime is 55 minutes
System returned to ROM by power-on
System image file is "flash:c2800nm-adventerprisek9-mz.151-4.M12a.bin"
Last reload type: Normal Reload
[output truncated]

Router1#verify flash:c2800nm-adventerprisek9-mz.151-4.M12a.bin
Verifying file integrity of flash:c2800nm-adventerprisek9-mz.151-4.M12a.bin.................................
............................................................................................................
[output truncated]
............................................................................................................
............................................................................................................
....Done!
Embedded Hash   MD5 : 136DE2A76AC52173C9D05C1837FA5403
Computed Hash   MD5 : 136DE2A76AC52173C9D05C1837FA5403
CCO Hash        MD5 : FCDAEB55B292534E97ECC29A394D35AA
Embedded hash verification successful.

Note that the embedded hash and computed hash should retun the same MD5 value, and the CCO hash should match the MD5 value listed on CCO or in the Bulk Hash File for that particular image file.

Repeat the above procedure for any other system image file located on the file systems. A comprehensive list of all files can be viewed by executing the following command:

dir all-filesystems

If any of the image file hashes show inconsistencies, copy the image file in question to a secure location if possible.


copy <location>:<system_image_filename.bin> ftp: 
Address or name of remote host []? <destination_ip>
Destination filename []? <destination_filename.bin>

It is highly recommended that a hash value be calculated on the copied system image file and compared to the hash value obtained on the platform to ensure that no errors were introduced during the file transfer process.

Submit all command output and any system images collected in this section to the relevant TAC SR and proceed to the next section of this document.


Step Four – ROMMON Upgrade Check

This procedure checks whether the ROM Monitor firmware of the device has been upgraded. Cisco is aware of a small number of incidents where a modified firmware image has been introduced in order to change device behavior or to circumnavigate certain platform or IOS license checks.

Execute the following commands in enable mode:


terminal length 0
show rom-monitor

Example 1 depicts a platform where the ROM Monitor HAS NOT been upgraded.


EXAMPLE 1:
----------
Router1# show rom-monitor 
ReadOnly ROMMON version:
System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2006 by cisco Systems, Inc.
No upgrade ROMMON programmed or not yet run
Currently running ROMMON from ReadOnly region
ROMMON from ReadOnly region is selected for next boot

Example 2 depicts a platform where the ROM Monitor HAS been upgraded.


EXAMPLE 2:
----------
Router1#show rom-monitor 
ReadOnly ROMMON version:
System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2006 by cisco Systems, Inc.
Upgrade ROMMON version:
System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2009 by cisco Systems, Inc.
Currently running ROMMON from Upgrade region
ROMMON from Upgrade region is selected for next boot

Submit all command or script output collected in this section to the relevant TAC SR and proceed to the next section of this document.


Step Five – Cisco IOS Core File/Memory Dump

CAUTION: This section contains commands that alter device configuration. Please ensure that you have the appropriate authorization to make changes to the platform in question prior to proceeding with this procedure.

This procedure outlines how to configure a Cisco IOS device to obtain a dump of platform memory.

To configure the appropriate dump parameters, enter the following commands in enable mode:

conf t
service timestamps debug datetime msec localtime 
service timestamps log datetime msec localtime 
service internal 
exception core-file <filename.bin> compress 
exception region-size 65536
exception dump <destination_ip_address>                                                                 
exception protocol ftp 
ip ftp username <username>       
ip ftp password <password>                                   
end

CAUTION: Initiating a core dump on an IOS device can be CPU intensive and may adversely affect traffic transitting the platform.

To initiate the core dump process, execute the following command in enable mode:

write core

The core dump may take some time to complete, depending on the amount of physical memory (RAM) installed on the device.

When the core dump process is complete, remove the core dump parameters as follows:

conf t
no service internal 
no exception core-file
no exception region-size
no exception dump
no exception protocol ftp
no ip ftp username 
no ip ftp password
end

It is highly recommended that hash values be calculated on the core files obtained in this section so that any errors introduced by subsequent copying or transmission can be reliably detected.

Submit all command output and core files collected in this section to the relevant TAC SR.


Step Six – Analysis With the verify Command

Obtaining a core dump, as outlined in step 4, is the preferred method of gathering information for analysis in Cisco IOS device forensics. However, there may be times when retrieving a core dump is not possible and this section provides an alternative method for obtaining a copy of the runtime image from the memory a Cisco IOS device using the following command:

verify /md5 system:memory/text

Note: Due to a bug in Cisco IOS Software, the verify /md5 system:memory/text command may cause a crash and reload on some platforms with x86 CPUs. Therefore, this command should not be issued on the following platforms unless the appropriate fixes have been applied:

  • Cisco Catalyst 6880-X Switch (CSCus44072)
  • Cisco 3900E Series Integrated Services Routers (CSCus44043)
  • Cisco 1000 Series Connected Grid Routers (CSCus44013)

Access the command line interface of the Cisco IOS device and issue the following command in enable mode:


verify /md5 system:memory/text

An example of this procedure is as follows:


Router1#verify /md5 system:memory/text
..............................................................................
..............................................................................
[output truncated]
..............................................................................
.........................................................................Done!
verify /md5 (system:memory/text) = a610fcadcc2cc7578f53d2990483ec47

Submit all command output collected in this section to the relevant TAC SR.


Acknowledgments

The author would like to thank all members of the Customer Experience Security Programs (CXSP) and Advanced Security Initiatives Group (ASIG) who provided their expertise and support during the writing of this document.