Linux Forensics

Linux forensics focuses on log analysis, user activity reconstruction, and identifying persistence mechanisms used by attackers. Key artifacts include bash history, cron jobs, and system logs.

Key Linux Artifacts

The first question on a Linux host is where the logs actually live. Debian and Ubuntu images are increasingly journald-only, and RHEL, Fedora and SUSE have used journald by default for years — on those hosts /var/log/auth.log, /var/log/secure and /var/log/syslog do not exist at all, and grepping them proves nothing. Check which subsystem is present before choosing a command.

Systemd Journal (journald)

The journal is a binary, indexed store, so you query it rather than grep it. Two facts matter more than the syntax: whether storage is persistent (/var/log/journal) or volatile (/run/log/journal, which dies on reboot), and that a journal directory copied from an image can be read offline with -D.

# Where is the journal, and is it persistent?
ls -ld /var/log/journal /run/log/journal
journalctl --list-boots

# SSH authentication straight from the journal - no text file involved
journalctl _COMM=sshd --since "2026-05-20" --until "2026-05-27" --no-pager
journalctl _COMM=sshd -g "Accepted|Failed password|Invalid user" --since "2026-05-20"

# One unit, newest first
journalctl -u sshd.service -r --no-pager
journalctl -u cron.service -u crond.service --since yesterday

# Structured export - keep the raw JSON, it is the evidence
journalctl -u sshd.service -o json-pretty --since "2026-05-20" > sshd-journal.json
journalctl -o export --since "2026-05-20" > journal.export

# Read a journal copied from a mounted image (offline)
journalctl -D /mnt/evidence/var/log/journal --no-pager --since "2026-05-20"

Text Log Files (rsyslog hosts)

When rsyslog is installed alongside journald, the same events also land in plain files — and those files are what log-shipping and most SIEM documentation assume. The path differs by distribution, so confirm the file exists before parsing it.

# Debian/Ubuntu authentication log (only if rsyslog is installed)
grep -E "Accepted|Failed password" /var/log/auth.log

# RHEL/CentOS/Fedora/SUSE authentication log
grep -E "Accepted|Failed password" /var/log/secure

# General system logs: syslog on Debian, messages on RHEL and SUSE
grep -i "error" /var/log/syslog
grep -i "error" /var/log/messages

Login Records and Shell History

These answer who logged in, from where, and what they typed. The wtmp, btmp and lastlog files are binary, so use the tools that read them rather than cat.

# Successful logins, failed logins, last login per account
last -F | head -50
lastb | head -50
lastlog

# Shell history and SSH material per user
/home/*/.bash_history
/home/*/.bashrc
/home/*/.ssh/authorized_keys
/home/*/.ssh/known_hosts
/root/.bash_history

Persistence Locations

Enumerate what is scheduled and what is enabled, rather than trusting the file names you remember from training. A cron entry the attacker added, a unit enabled with systemctl enable, and a line in /etc/ld.so.preload are three different mechanisms with three different checks.

# Scheduled work
cat /etc/crontab
ls -la /etc/cron.*/
for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2>/dev/null; done
ls -la /var/spool/cron/crontabs/

# Enabled units, and units whose files changed recently
systemctl list-unit-files --state=enabled
find /etc/systemd/system -type f -mtime -30 -ls

# Classic files
cat /etc/rc.local
ls -la /etc/init.d/
cat /etc/ld.so.preload

# Recent file modifications across the file system (noisy - scope it)
find / -xdev -mtime -7 -type f 2>/dev/null
find / -xdev -ctime -1 -type f 2>/dev/null

Container and Kubernetes Evidence

A container is the least durable object in the estate, and the reflex remediation — delete the pod, let the deployment recreate it — destroys the writable layer. Container forensics is a race with a scope problem attached: decide which question the evidence must answer, take the node-level copy before anyone drains anything, and treat the control plane's records as the durable half of the case.

Why the Pod Is Usually Gone

Four ordinary mechanisms remove a container before an analyst hears about it: a rollout replaces every pod in a deployment, a failed liveness probe restarts the container in place, the kubelet evicts pods under node pressure, and image garbage collection deletes images it believes are unused. Each discards the container's writable layer — an overlayfs upper directory on the node — and any emptyDir volume the pod mounted. The image is reproducible from the registry; the layer holding what the attacker wrote is not.

Managed clusters add a scoping constraint before any of that. On EKS, GKE and AKS you do not get the etcd snapshot, and depending on how the cluster was created you may not get node access at all. Establish what the platform lets you reach in the first ten minutes: an acquisition plan that assumes node access is worthless on a cluster that denies it.

Live Capture on the Node

kubectl debug node starts a privileged pod that mounts the host root at /host; from inside it, the container runtime's own CLI sees every container on the node, including the exited ones a kubectl get pods listing no longer shows. Cordon the node first and do not drain it: draining evicts the pods you came for.

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -40
kubectl cordon ip-10-0-12-34.eu-west-1.compute.internal

kubectl debug node/ip-10-0-12-34.eu-west-1.compute.internal -it --image=busybox:1.36

# From inside the debug pod - the host filesystem is mounted at /host
chroot /host
crictl ps --all
crictl inspect 3f2b1c9d8e7a > /host/tmp/ir-3f2b1c9d8e7a.json
crictl inspecti sha256:9f8e7d6c5b4a39281706f5e4d3c2b1a09876543210fedcba9876543210abcdef
exit
kubectl cp default/node-debugger-ip-10-0-12-34:/host/tmp/ir-3f2b1c9d8e7a.json ./ir-3f2b1c9d8e7a.json

crictl inspect returns the runtime's view of one container: the full spec, mounts, environment, image reference and the bundle path the runtime created for it. The writable layer sits in the snapshotter's store; for containerd, the snapshot key from ctr -n k8s.io containers info <id> resolves to the overlay directories, and that is the tree you copy while the container is still running. An ephemeral container joins the target's namespaces without restarting it — kubectl debug -it <pod> --target=<container> --image=busybox:1.36 — but it appends to the pod spec: record it as an alteration, not a harmless read.

The Image, by Digest

A tag is a mutable pointer and a digest is a content address, so the only defensible reference to an image is the digest. Take it from the registry rather than the node: the registry holds the artefact that shipped, while the node holds whatever it last pulled. Export to an OCI layout rather than a flattened filesystem, because flattening loses the layer boundaries that tell you which layer added the file you care about.

crane digest registry.example.com/payments/api:1.4.2
crane manifest registry.example.com/payments/api@sha256:9f8e7d6c5b4a39281706f5e4d3c2b1a09876543210fedcba9876543210abcdef

mkdir -p /evidence/api-image
skopeo copy --all \
  docker://registry.example.com/payments/api@sha256:9f8e7d6c5b4a39281706f5e4d3c2b1a09876543210fedcba9876543210abcdef \
  oci:/evidence/api-image:1.4.2
sha256sum /evidence/api-image/index.json > /evidence/api-image/index.json.sha256

The failure mode this creates: if the pipeline overwrote the tag and the registry has already garbage-collected the old manifest, the digest in your notes resolves to nothing. That is why immutable-tag or retention settings belong in the readiness review rather than the post-incident report.

etcd Is the Cluster's Ground Truth

Every object the API server ever accepted is a key in etcd, including the pods that no longer exist as running containers, the secrets they mounted and the RBAC that let them. On a self-managed control plane a snapshot is the single most valuable artefact in the cluster, and it is taken with the same client certificate the API server uses. Restore it into a scratch cluster with no network path to production, never into the live one.

etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /evidence/etcd-2026-05-26.db

etcdctl snapshot status /evidence/etcd-2026-05-26.db -w table
sha256sum /evidence/etcd-2026-05-26.db > /evidence/etcd-2026-05-26.db.sha256

Two caveats decide how much the snapshot proves. If encryption at rest uses a KMS provider, secret values in the snapshot are ciphertext and the key lives outside the cluster, so the snapshot alone will not reveal a stolen credential. And it records the API server's accepted state, not the node's — a file written into a container never appears in etcd, only the pod spec that described the mount.

Audit Logs Record Intent, Not Payload

The API server audit log has four levels, and the level chosen for a resource decides whether you will ever see what was sent. None records nothing, Metadata records who, what, when and the response status, Request adds the request body, and RequestResponse adds the response body too. Policies in the wild usually log at Metadata for almost everything, because RequestResponse on every resource is enormous and writes secrets into the log. The consequence: an exec into a pod appears with the requesting identity and the target container, and the commands typed inside it do not appear at all.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
  - level: Metadata
    omitStages: ["RequestReceived"]

On a self-managed control plane retention is set by --audit-log-maxage, --audit-log-maxbackup and --audit-log-maxsize; on managed clusters it is whatever the platform's log export keeps, commonly 30 days unless you shipped it elsewhere. Capture the policy file with the logs — it defines what the absence of an event means.

What Survives Deletion

Enough survives to answer most questions, provided somebody looks on the node rather than in the API:

• Container stdout on the node. /var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log holds the CRI-formatted stream with timestamps and outlives the pod until rotation.
• Runtime state. containerd keeps the writable layer under its snapshotter directory and Docker under /var/lib/docker/overlay2/<id>/diff; both vanish at the next garbage collection. Locally built images that never reached the registry survive here too.
• The kubelet's own log in journald, with image pulls, container starts and restarts.
• Network records — CNI or cloud flow logs keyed to the node's interface, and DNS query logs if the resolver kept them.
• The log pipeline: whatever your collector shipped is now the only copy, with a retention of its own.
• The control-plane audit record of the delete itself, which anchors the identity and the time.

Detections Are Evidence Only If You Shipped Them

A runtime security tool is a sensor, not a recorder. Falco writes alerts to stdout, syslog or a gRPC stream, and nothing is persisted unless something downstream keeps it; Tetragon emits structured JSON events and can write them to a file, which still needs rotation and shipping. Both are worth having for one reason that outlives the pod: the alert carries the process arguments, the parent process and the container identity, so an alert record can reconstruct an execution that the container's own filesystem no longer contains.

Record the rule set alongside the alert — the rules file, its version and a hash. The same event means different things under different rules, so "no alert was raised" is a statement about the ruleset that was loaded. A disabled rule or a dropped event source is a finding an attacker with cluster admin can produce without touching a workload.