Show The Graduate Center Menu

Advanced network security and malware analysis


Advanced network security and malware analysis is an advanced course for individuals interested in the theory and practice of network security.
This course will study approaches, mechanisms, and tools used to make networks and software systems more secure against malware, based on a survey of recent and seminal research papers. We will motivate the study by discussing common software security dangers (e.g., buffer overflow attacks, control-hijacking), common network security dangers (malware, botnets, APTs), and malware analysis tools (e.g. Binary Analysis Platform). The majority of the course will be divided into three main
modules: architectural approaches to building secure software (e.g., confinement, virtual machines, trusted computing); software analysis (e.g., static analysis and testing, model checking, dynamic analysis, code cloning) for finding software flaws as well as analyzing malicious code; and network detection techniques for network-based malware.

Learning Goals

    •    Understand the state of the art in control-hijacking and associated defenses in software systems.
    •    Understand the specific security architectures for system isolation and analysis.
    •    Understand the strengths and limitations of various methods of software analysis, and their application of vulnerability discovery and verification of security properties, as also applied to malware analysis.
    •    Understand the state of the art in malicious network activity detection techniques.
Reference books and web resources
    •    Jelena Mirkovic, Sven Dietrich, David Dittrich, and Peter Reiher.
Internet Denial of Service: Attack and Defense Mechanisms, Prentice Hall PTR, 2004.
    •    Applied Crypto
    •    Additional online up-to-date materials will be provided as time
progresses, mostly from research conferences in systems-oriented security.

Tentative topics (Part 1: 2.5-3 weeks; Part 2: 4 weeks; Part 3: 4.5-5 weeks)

    •    Part 1
    ◦    Ethics in Cybersecurity
    ▪    Menlo Report: guidelines for ethical cybersecurity research
    ◦    Software basics
    ▪    Review of the basics of code generation and trust.
    ◦    Attacks
    ▪    Buffer overflows (stack, heap, etc.)
    ◦    Defenses
    ▪    Separation, memory protection
    ▪    Virtual machines, sandboxing
    ▪    Isolation and confinement.
    ◦    Control-flow integrity
    ◦    Enforcing security properties at run-time
    •    Part 2
    ◦    Cryptography overview
    ◦    Trusted Computing
    ▪    Software security architectures
    ▪    Static analysis of C programs
    ▪    Dynamic analysis
    ▪    Software model checking
    ▪    Building verifiable systems
    •    Part 3
    ◦    Malicious code analysis (Binary Analysis)
    ▪    Sandboxed analysis
    ▪    VM introspection
    ▪    Malicious code classification
    ◦    Botnet analysis & detection
    ▪    Activity-based (DDoS, click fraud)
    ▪    Command-and-Control (C&C) based
    ▪    Side-channel detection
    ▪    Correlation analysis
    ▪    Topologies
    ◦    Advanced Persistent Threats
    ▪    Exfiltration
    ▪    C&C
    ◦    Moving Target Defense



Students will be evaluated based on small projects, two papers, and class participation. The projects will provide students with practical experience with the tools and mechanisms studied in class. Students will work on the projects in groups of two or three, and the projects will be evenly spaced over the course of the semester.


There will be weekly in-class discussion
    •    25%x3 projects
    •    5% class participation
    •    20% for writing 2 research papers


All assignments must be your own work and submitted on time. Late submission without prior permission will not be graded.