Last reviewed

Intermediate

Malware Analysis Lab Setup

Build a safe, isolated environment for analyzing malicious software. Learn to set up sandboxed VMs, network isolation, and analysis tools without risking your host system.

Safety First

Malware analysis carries real risks. Always use isolated VMs, disable network access to your LAN, and never analyze samples on your host machine. When in doubt, use a dedicated physical machine with no network connection.

Sample Handling Rules

Store samples in encrypted archives, record source and hash values before analysis, disable shared clipboard and shared folders, and keep the chain of custody outside the disposable analysis VM.

Lab Runbook

Use this page as a controlled lab build, not a production hardening guide. Validate isolation before running exercises and write down the cleanup command before starting.

Critical risk Intermediate 2-4 hr

Plan

8-24 GB; 100 GB+. Free. Isolation: Internal network only; no route to host LAN or internet.

Build

  • - Analysis VM
  • - Victim VM snapshot
  • - INetSim services

Validate

  • - Victim has no internet route
  • - DNS resolves to INetSim
  • - Packet capture sees simulated traffic

Exercise

Run only the exercises tied to this lab and save screenshots, command output, logs, and timestamps outside disposable VMs.

Clean Up

  • - Revert victim snapshot after every sample
  • - Store samples in password-protected archives
  • - Purge shared folders and clipboard settings

Lab Architecture

HOST MACHINE
Isolated Virtual Network
(No Internet / Host-Only or Internal Network)
Analysis VM
(REMnux)
  • • Ghidra
  • • x64dbg
  • • Wireshark
  • • YARA
Victim VM
(Windows)
  • • Snapshots
  • • Procmon
  • • API Mon
  • • Autoruns
INetSim VM
(Services)
  • • Fake DNS
  • • Fake HTTP
  • • Fake SMTP
  • • Fake IRC
Isolated Switch

Required Virtual Machines

🔬 Analysis VM (REMnux)

  • OS: REMnux (Ubuntu-based)
  • RAM: 4-8GB
  • Disk: 60GB+
  • Purpose: Static analysis, network capture
remnux.org →

🎯 Victim VM (Windows)

  • OS: Windows 10/11
  • RAM: 4-8GB
  • Disk: 60GB+
  • Purpose: Execute & monitor malware

Use evaluation ISOs from Microsoft

🌐 INetSim VM

  • OS: Debian/Ubuntu minimal
  • RAM: 1-2GB
  • Disk: 20GB
  • Purpose: Simulate internet services
inetsim.org →

Network Isolation Setup

Critical

NEVER connect malware analysis VMs to your real network. Use Host-Only or Internal networking only. Malware can spread to other machines on your network.

VirtualBox Configuration

bash 
# Create host-only network in VirtualBox
VBoxManage hostonlyif create
VBoxManage hostonlyif ipconfig vboxnet0 --ip 10.0.0.1 --netmask 255.255.255.0

# Disable DHCP on host-only network (manual IPs only)
VBoxManage dhcpserver remove --netname HostInterfaceNetworking-vboxnet0

# Configure VM to use host-only adapter
VBoxManage modifyvm "MalwareAnalysis" --nic1 hostonly --hostonlyadapter1 vboxnet0
# Create host-only network in VirtualBox
VBoxManage hostonlyif create
VBoxManage hostonlyif ipconfig vboxnet0 --ip 10.0.0.1 --netmask 255.255.255.0

# Disable DHCP on host-only network (manual IPs only)
VBoxManage dhcpserver remove --netname HostInterfaceNetworking-vboxnet0

# Configure VM to use host-only adapter
VBoxManage modifyvm "MalwareAnalysis" --nic1 hostonly --hostonlyadapter1 vboxnet0

VMware Configuration

bash 
# Edit vmnetcfg or use Virtual Network Editor
# Create custom VMnet (e.g., VMnet2)
# Settings:
#   - Host-only
#   - Subnet: 10.0.0.0/24
#   - DHCP disabled
#   - "Connect a host virtual adapter" UNCHECKED (important!)

# Assign VMs to this custom network in VM settings
# Edit vmnetcfg or use Virtual Network Editor
# Create custom VMnet (e.g., VMnet2)
# Settings:
#   - Host-only
#   - Subnet: 10.0.0.0/24
#   - DHCP disabled
#   - "Connect a host virtual adapter" UNCHECKED (important!)

# Assign VMs to this custom network in VM settings

Static IP Configuration

VM IP Address Gateway DNS
INetSim (Gateway) 10.0.0.1 - -
Analysis VM 10.0.0.10 10.0.0.1 10.0.0.1
Victim VM 10.0.0.100 10.0.0.1 10.0.0.1

INetSim Configuration

INetSim simulates common internet services, tricking malware into thinking it has internet access while you capture all its communications.

bash 
# Install INetSim
sudo apt update && sudo apt install inetsim

# Edit configuration
sudo nano /etc/inetsim/inetsim.conf

# Key settings to modify:
service_bind_address    10.0.0.1
dns_default_ip          10.0.0.1
dns_default_hostname    inetsim

# Enable services
start_service dns
start_service http
start_service https
start_service smtp
start_service pop3
start_service irc

# Start INetSim
sudo inetsim
# Install INetSim
sudo apt update && sudo apt install inetsim

# Edit configuration
sudo nano /etc/inetsim/inetsim.conf

# Key settings to modify:
service_bind_address    10.0.0.1
dns_default_ip          10.0.0.1
dns_default_hostname    inetsim

# Enable services
start_service dns
start_service http
start_service https
start_service smtp
start_service pop3
start_service irc

# Start INetSim
sudo inetsim

Essential Analysis Tools

Static Analysis

  • Ghidra: NSA's reverse engineering framework
  • IDA Free: Interactive disassembler
  • PE-bear: PE file analyzer
  • DIE: Detect It Easy - packer detection
  • strings/FLOSS: String extraction
  • YARA: Pattern matching rules

Dynamic Analysis

  • x64dbg: Windows debugger
  • Process Monitor: System activity monitoring
  • Process Hacker: Advanced task manager
  • Wireshark: Network traffic capture
  • Regshot: Registry comparison
  • API Monitor: API call logging

Victim VM Configuration

powershell 
# Analysis VM only: reduce host controls after snapshotting and isolating the network
Set-MpPreference -DisableRealtimeMonitoring $true
Set-MpPreference -SubmitSamplesConsent NeverSend

  # Analysis VM only: disable Windows Firewall after confirming no route to your LAN
netsh advfirewall set allprofiles state off

# Install analysis tools via Chocolatey
Set-ExecutionPolicy Bypass -Scope Process -Force
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

# Install tools (run before isolating network)
choco install -y procexp procmon autoruns wireshark x64dbg.portable

# Take a CLEAN SNAPSHOT before any malware execution
# Name it: "Clean-Baseline-YYYYMMDD"
# Analysis VM only: reduce host controls after snapshotting and isolating the network
Set-MpPreference -DisableRealtimeMonitoring $true
Set-MpPreference -SubmitSamplesConsent NeverSend

  # Analysis VM only: disable Windows Firewall after confirming no route to your LAN
netsh advfirewall set allprofiles state off

# Install analysis tools via Chocolatey
Set-ExecutionPolicy Bypass -Scope Process -Force
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

# Install tools (run before isolating network)
choco install -y procexp procmon autoruns wireshark x64dbg.portable

# Take a CLEAN SNAPSHOT before any malware execution
# Name it: "Clean-Baseline-YYYYMMDD"

Snapshot Strategy

Always take a clean snapshot before analyzing each sample. After analysis, revert to the clean state. Never accumulate malware executions on a single snapshot.

Alternative: FLARE VM

Mandiant's FLARE VM is a pre-configured Windows-based malware analysis distribution with 140+ tools pre-installed.

powershell 
# Start with fresh Windows 10+ VM
# Analysis VM only: follow FLARE VM prerequisites after snapshotting and isolating the host
# Run PowerShell as Administrator:

# Download and run FLARE VM installer
(New-Object net.webclient).DownloadFile(
  'https://raw.githubusercontent.com/mandiant/flare-vm/main/install.ps1',
  "$env:TEMP\install.ps1"
)
Unblock-File "$env:TEMP\install.ps1"
Set-ExecutionPolicy Unrestricted -Force
& "$env:TEMP\install.ps1"

# Follow prompts - installation takes 1-2 hours
# Take snapshot after installation completes
# Start with fresh Windows 10+ VM
# Analysis VM only: follow FLARE VM prerequisites after snapshotting and isolating the host
# Run PowerShell as Administrator:

# Download and run FLARE VM installer
(New-Object net.webclient).DownloadFile(
  'https://raw.githubusercontent.com/mandiant/flare-vm/main/install.ps1',
  "$env:TEMP\install.ps1"
)
Unblock-File "$env:TEMP\install.ps1"
Set-ExecutionPolicy Unrestricted -Force
& "$env:TEMP\install.ps1"

# Follow prompts - installation takes 1-2 hours
# Take snapshot after installation completes

Alternative: CAPE Sandbox

CAPE (Config And Payload Extraction) is a self-hosted automated malware analysis sandbox, the actively maintained successor to Cuckoo Sandbox. It automatically extracts configs, payloads, and IOCs from malware samples.

bash 
# CAPE requires a dedicated Ubuntu 22.04 host or VM
# Full installation guide: https://capev2.readthedocs.io

# Quick start with Docker (for evaluation)
git clone https://github.com/kevoreilly/CAPEv2.git
cd CAPEv2

# Use the provided installer script
sudo ./installer/cape2.sh base
sudo ./installer/cape2.sh cape

# Configure analysis VMs in conf/virtualbox.conf or conf/kvm.conf
# Submit samples via web UI at https://localhost:8443
# CAPE requires a dedicated Ubuntu 22.04 host or VM
# Full installation guide: https://capev2.readthedocs.io

# Quick start with Docker (for evaluation)
git clone https://github.com/kevoreilly/CAPEv2.git
cd CAPEv2

# Use the provided installer script
sudo ./installer/cape2.sh base
sudo ./installer/cape2.sh cape

# Configure analysis VMs in conf/virtualbox.conf or conf/kvm.conf
# Submit samples via web UI at https://localhost:8443

Cloud-Based Triage (Quick Analysis)

Before spinning up your full lab, use cloud sandboxes for quick initial triage:

ANY.RUN

Interactive cloud sandbox. Watch malware execute in real-time in a browser-based VM. Free tier available.

any.run →

Triage (Hatching)

Automated sandbox with YARA rules, behavioral analysis, and config extraction. Free community tier.

tria.ge →

Joe Sandbox

Deep analysis with MITRE ATT&CK mapping. Cloud and on-premise options. Free community tier.

joesandbox.com →

When to Use Cloud vs. Local

Cloud sandboxes are great for quick triage and getting initial behavioral data. Use your local lab for deep analysis, debugging, unpacking, and when you need full control of the environment.

Cloud Sandbox Privacy

Cloud sandboxes may share submissions, metadata, screenshots, network indicators, and extracted files with other users or partners depending on account type. Do not upload client samples, sensitive documents, or unreleased incident artifacts unless your authorization and data-handling rules allow it.

Malware Sample Sources

Handle With Care

These sources contain REAL malware. Only download samples to isolated analysis VMs. Never execute samples outside your sandboxed environment.

Free Sources

Practice Samples

Analysis Workflow

1

Prepare Environment

Start from clean snapshot, verify network isolation, start INetSim and packet capture

2

Static Analysis

Hash sample, check VT, extract strings, identify packers, review imports/exports

3

Dynamic Analysis

Start monitoring tools, execute sample, observe behavior, capture network traffic

4

Deep Analysis

Disassemble in Ghidra, debug with x64dbg, unpack if needed, extract IOCs

5

Document & Clean Up

Export PCAP, save IOCs, document findings, REVERT TO CLEAN SNAPSHOT

Quick Reference Commands

bash 
# Hash a file (PowerShell)
Get-FileHash -Algorithm SHA256 sample.exe

# Hash a file (Linux)
sha256sum sample.exe
md5sum sample.exe

# Extract strings (Linux)
strings -a sample.exe | less
floss sample.exe  # Better string extraction

# Check PE headers
pecheck sample.exe
pefile-parse sample.exe

# Start Wireshark capture
wireshark -i eth0 -k -w capture.pcap

# YARA scan
yara -r rules/ sample.exe
# Hash a file (PowerShell)
Get-FileHash -Algorithm SHA256 sample.exe

# Hash a file (Linux)
sha256sum sample.exe
md5sum sample.exe

# Extract strings (Linux)
strings -a sample.exe | less
floss sample.exe  # Better string extraction

# Check PE headers
pecheck sample.exe
pefile-parse sample.exe

# Start Wireshark capture
wireshark -i eth0 -k -w capture.pcap

# YARA scan
yara -r rules/ sample.exe

Safety Reminder

Malware analysis is inherently dangerous. Always maintain strict isolation, use dedicated hardware when possible, and never underestimate a sample's capabilities. Some malware can detect and escape virtual environments.

Troubleshooting FAQ

Malware sample detected and quarantined by host AV
  • Password-protect sample ZIPs (standard: infected)
  • Add exclusions in host AV for your analysis VM shared folders
  • Transfer samples via SCP/SFTP directly into the isolated VM
  • Consider using a dedicated physical machine for malware analysis
Sample not executing in the VM
  • Check architecture (x86 vs x64) — use file sample.exe on Linux
  • Some malware detects VMs — disable VM artifacts: remove VMware Tools, change MAC prefix
  • Missing dependencies: install Visual C++ runtimes and .NET frameworks
  • Try running as Administrator with Defender/AV disabled
INetSim not capturing DNS/HTTP traffic
  • Ensure victim VM DNS points to the REMnux/INetSim IP
  • Verify INetSim is running: sudo inetsim (check for binding errors)
  • Check firewall rules on the REMnux VM — ports 53, 80, 443 must be open
  • Confirm both VMs are on the same isolated host-only network

Go Deeper After Setup

Malware Analysis Move from lab setup into triage and behavior analysis. Reverse Engineering Use the same lab for deeper unpacking and static analysis.

Operational Safety Baseline

Apply these rules before running any lab command on this page.

  • Work only on systems you own or have explicit authorization to test.
  • Keep vulnerable services off your home LAN and off public interfaces.
  • Take clean snapshots before every exercise and before every vulnerable configuration change.
  • Use dedicated cloud accounts, subscriptions, and projects with billing alerts before deployment.
  • Write down the teardown command before you run the setup command.

Validation Checkpoints

  • -Victim has no internet route
  • -DNS resolves to INetSim
  • -Packet capture sees simulated traffic
  • -Snapshots are immutable before detonation

Cleanup And Rollback

  • -Revert victim snapshot after every sample
  • -Store samples in password-protected archives
  • -Purge shared folders and clipboard settings

Practice Next

Malware Analysis Reverse Engineering

Related Guides

Malware Analysis Guide

Static and dynamic analysis techniques

DFIR

Digital forensics and incident response

Kali Setup

Configure your analysis machine

Threat Intelligence

IOC extraction and analysis

Reverse Engineering

Binary analysis and disassembly

Binary Exploitation

Understanding binary internals