Skip to main content

Efficient and Effective High Speed Network Logging for Digital Forensics

Researcher: Dave Andersen

Research Area: Next Generation Secure and Available Networks


Scope: This project seeks to create software systems infrastructure for, in a cost and resource-effective manner, recording, indexing, and supporting rich queries against a full record of all network traffic sent to and from an enterprise, with such logs being kept for a week or longer. The primary motivation for creating this capability is post-facto forensics: After an intrusion or other activity is discovered, being able to “go back in time” and audit all traffic to and from a compromised host, to an application subsequently found to be vulnerable, or with an outside host found to have launched attacks. The techniques have further application to network management and monitoring, research, and to storage and archiving systems.

Outcomes: The successful completion of the research will result in a new capability for monitoring and managing enterprise networks. At present, the hardware requirements for the extensive logging we propose are untenable: The storage and memory costs are high, and using traditional approaches (e.g., databases) for supporting queries against a tens of terabytes dataset require solutions costing in the hundreds of thousands of dollars. We believe we can create the capability to monitor a large enterprise, such as Carnegie Mellon, using a handful of commodity machines, while enabling efficient retrieval of flows that an operator or analysis program decides are suspicious.