14 min read

Expert

Race Conditions Remediation

Risk Severity
High
Fix Effort
Medium
Est. Time
3-6 hours
Reference
A01:2025 CWE-362

Race conditions occur when multiple threads or processes access shared resources concurrently without proper synchronization, leading to inconsistent state, privilege escalation, or data corruption.

Critical Impact

Race conditions can bypass security checks, corrupt data, cause double-spending in payment systems, and lead to privilege escalation. They're often difficult to detect and reproduce.

Race Condition Attack Flow

Understanding Race Conditions

Common Race Condition Types

  • • TOCTOU: Time-of-Check-Time-of-Use
  • • Database races: Concurrent transactions
  • • File system races: Symlink attacks
  • • State transitions: Order-dependent operations
  • • Double-spend: Balance check race

Attack Scenario

  1. Thread 1 checks if file exists
  2. Thread 2 creates malicious symlink
  3. Thread 1 opens file (now follows symlink)
  4. Attacker reads/writes sensitive file

TOCTOU Prevention

Best Practice

Eliminate the window between check and use by performing atomic operations. Use "try and fail" instead of "check then act".

Vulnerable File Access (TOCTOU)

// ❌ VULNERABLE: Check-then-act pattern
if (fs.existsSync(filename)) {
    // Race window here! File could be deleted or replaced
    const content = fs.readFileSync(filename);
    processFile(content);
}

// ✅ SECURE: Try-and-catch pattern
try {
    const content = fs.readFileSync(filename);
    processFile(content);
} catch (err) {
    // Handle missing file
}

// ✅ SECURE: Atomic file operations (Node.js)
const fd = fs.openSync(filename, 'r', 0o666);
try {
    const content = fs.readFileSync(fd);
    processFile(content);
} finally {
    fs.closeSync(fd);
}

Python Safe File Operations

# ❌ VULNERABLE: Separate check and open
import os
if os.path.exists(filename):
    with open(filename) as f:  # Race window!
        data = f.read()

# ✅ SECURE: Try to open directly
try:
    with open(filename, 'r') as f:
        data = f.read()
except FileNotFoundError:
    # Handle missing file
    pass

# ✅ SECURE: Use exclusive creation flag
import os
try:
    # O_CREAT | O_EXCL ensures atomic creation
    fd = os.open(filename, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
    with os.fdopen(fd, 'w') as f:
        f.write(sensitive_data)
except FileExistsError:
    # File already exists - safe failure
    pass

Database Race Conditions

Vulnerable Balance Check

-- ❌ VULNERABLE: Separate read and update
-- Thread 1: SELECT balance = 100
-- Thread 2: SELECT balance = 100
-- Thread 1: UPDATE balance = 100 - 60 = 40
-- Thread 2: UPDATE balance = 100 - 50 = 50  ← Wrong! Should be -10

-- Query 1: Check balance
SELECT balance FROM accounts WHERE user_id = 123;

-- Race window here!

-- Query 2: Deduct money
UPDATE accounts 
SET balance = balance - 50 
WHERE user_id = 123;

Secure: Atomic Operations

-- ✅ SECURE: Single atomic transaction with constraint
UPDATE accounts 
SET balance = balance - 50 
WHERE user_id = 123 
  AND balance >= 50;  -- Check happens atomically

-- Check affected rows
IF @@ROWCOUNT = 0 THEN
    RAISE 'Insufficient funds';
END IF;

-- ✅ SECURE: Use database transactions with serializable isolation
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
    
    SELECT balance FROM accounts WHERE user_id = 123 FOR UPDATE;
    
    -- Row is now locked, no other transaction can modify it
    
    IF balance >= 50 THEN
        UPDATE accounts SET balance = balance - 50 WHERE user_id = 123;
    ELSE
        ROLLBACK;
        RAISE 'Insufficient funds';
    END IF;
    
COMMIT;

Application Synchronization

Python Threading Locks

import threading

# ❌ VULNERABLE: No synchronization
balance = 1000

def withdraw(amount):
    global balance
    if balance >= amount:
        # Race window!
        balance -= amount
        return True
    return False

# ✅ SECURE: Use lock for critical section
balance = 1000
balance_lock = threading.Lock()

def withdraw_safe(amount):
    global balance
    with balance_lock:
        if balance >= amount:
            balance -= amount
            return True
        return False

# ✅ SECURE: Use thread-safe data structures
from queue import Queue
from threading import Semaphore

# Semaphore for resource limiting
max_connections = Semaphore(10)

def process_request():
    with max_connections:
        # Only 10 threads can be here at once
        handle_connection()

Java Synchronized Methods

// ❌ VULNERABLE: Non-atomic check-then-act
private int balance = 1000;

public boolean withdraw(int amount) {
    if (balance >= amount) {  // Race!
        balance -= amount;
        return true;
    }
    return false;
}

// ✅ SECURE: Synchronized method
private int balance = 1000;

public synchronized boolean withdraw(int amount) {
    if (balance >= amount) {
        balance -= amount;
        return true;
    }
    return false;
}

// ✅ SECURE: Use concurrent collections
import java.util.concurrent.atomic.AtomicInteger;

private AtomicInteger balance = new AtomicInteger(1000);

public boolean withdraw(int amount) {
    while (true) {
        int current = balance.get();
        if (current < amount) {
            return false;
        }
        int next = current - amount;
        if (balance.compareAndSet(current, next)) {
            return true;  // Success!
        }
        // CAS failed, retry
    }
}

// ✅ SECURE: ReentrantLock with conditions
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

private final Lock lock = new ReentrantLock();

public boolean withdraw(int amount) {
    lock.lock();
    try {
        if (balance >= amount) {
            balance -= amount;
            return true;
        }
        return false;
    } finally {
        lock.unlock();
    }
}

Distributed System Races

Redis Distributed Locks

// ✅ SECURE: Redis distributed lock with Redlock algorithm
const redis = require('redis');
const { promisify } = require('util');

async function acquireLock(client, lockKey, ttl = 5000) {
    const lockValue = crypto.randomUUID();
    
    // SET with NX (only if not exists) and PX (TTL in ms)
    const result = await client.set(
        lockKey,
        lockValue,
        'NX',
        'PX',
        ttl
    );
    
    if (result === 'OK') {
        return lockValue;  // Lock acquired
    }
    return null;  // Lock held by another process
}

async function releaseLock(client, lockKey, lockValue) {
    // Use Lua script for atomic check-and-delete
    const script = `
        if redis.call("get", KEYS[1]) == ARGV[1] then
            return redis.call("del", KEYS[1])
        else
            return 0
        end
    `;
    
    await client.eval(script, 1, lockKey, lockValue);
}

// Usage
async function processWithLock() {
    const lockValue = await acquireLock(redisClient, 'resource:123');
    if (!lockValue) {
        throw new Error('Could not acquire lock');
    }
    
    try {
        // Critical section
        await performTransaction();
    } finally {
        await releaseLock(redisClient, 'resource:123', lockValue);
    }
}

Database-Level Pessimistic Locking

-- ✅ SECURE: Row-level locking with SELECT FOR UPDATE
BEGIN;

-- Lock the row exclusively
SELECT * FROM inventory 
WHERE product_id = 123 
FOR UPDATE;

-- No other transaction can modify this row until we commit

UPDATE inventory 
SET quantity = quantity - 1 
WHERE product_id = 123;

COMMIT;

-- ✅ SECURE: Named advisory locks (PostgreSQL)
SELECT pg_advisory_lock(123456);

-- Critical section - only one session can hold this lock

SELECT pg_advisory_unlock(123456);

-- ✅ SECURE: Optimistic locking with version column
UPDATE products 
SET 
    quantity = quantity - 1,
    version = version + 1
WHERE 
    product_id = 123 
    AND version = @expected_version;

-- Check affected rows
IF @@ROWCOUNT = 0 THEN
    RAISE 'Concurrent modification detected';
END IF;

State Transition Safety

// ❌ VULNERABLE: Unprotected state transitions
class Order {
    constructor() {
        this.status = 'pending';
    }
    
    approve() {
        this.status = 'approved';  // Race!
    }
    
    cancel() {
        this.status = 'cancelled';  // Could overwrite approval!
    }
}

// ✅ SECURE: Atomic state transitions with validation
class OrderSafe {
    constructor() {
        this.status = 'pending';
        this.version = 0;
    }
    
    async transitionTo(newStatus, expectedVersion) {
        // Database enforces version check atomically
        const result = await db.query(
            `UPDATE orders 
             SET status = $1, version = version + 1
             WHERE id = $2 
               AND status = $3
               AND version = $4`,
            [newStatus, this.id, this.status, expectedVersion]
        );
        
        if (result.rowCount === 0) {
            throw new Error('Invalid state transition or concurrent modification');
        }
        
        this.status = newStatus;
        this.version++;
    }
    
    async approve() {
        if (this.status !== 'pending') {
            throw new Error('Can only approve pending orders');
        }
        await this.transitionTo('approved', this.version);
    }
}

// ✅ SECURE: State machine with explicit transitions
const validTransitions = {
    'pending': ['approved', 'cancelled'],
    'approved': ['shipped'],
    'shipped': ['delivered'],
    'cancelled': [],  // Terminal state
    'delivered': []   // Terminal state
};

function canTransition(currentState, newState) {
    return validTransitions[currentState]?.includes(newState) || false;
}

Testing Verification

Testing Challenge

Race conditions are notoriously difficult to test because they're timing-dependent. Use stress testing, thread analysis tools, and code review to identify vulnerabilities.

Concurrent Testing (Python)

import threading
import time

# Test for race conditions with concurrent threads
def test_concurrent_withdrawal():
    balance = Account(1000)
    threads = []
    
    # Try to withdraw $600 twice simultaneously (should fail once)
    def withdraw_thread():
        result = balance.withdraw(600)
        print(f"Withdrawal {'succeeded' if result else 'failed'}")
    
    # Launch multiple threads
    for _ in range(2):
        t = threading.Thread(target=withdraw_thread)
        threads.append(t)
        t.start()
    
    # Wait for completion
    for t in threads:
        t.join()
    
    final_balance = balance.get_balance()
    print(f"Final balance: {final_balance}")
    
    # Should be either 400 (one succeeded) or 1000 (both failed)
    # Should NEVER be -200 (both succeeded - race condition!)
    assert final_balance in [400, 1000], "Race condition detected!"

# Stress test with many threads
def stress_test_race_condition():
    account = Account(10000)
    threads = []
    
    def worker():
        for _ in range(100):
            account.withdraw(1)
            time.sleep(0.0001)  # Increase race window
    
    # 10 threads, 100 withdrawals each = 1000 total
    for _ in range(10):
        t = threading.Thread(target=worker)
        threads.append(t)
        t.start()
    
    for t in threads:
        t.join()
    
    expected_balance = 10000 - 1000
    actual_balance = account.get_balance()
    
    print(f"Expected: {expected_balance}, Actual: {actual_balance}")
    assert actual_balance == expected_balance, "Race condition detected!"

Testing Tools

Race Condition Detection Tools

  • • ThreadSanitizer (TSan): Detects data races in C/C++/Go
  • • Java Concurrency Stress Tests: jcstress framework
  • • Python threading analysis: pytest-timeout, pytest-xdist
  • • Static analysis: FindBugs, SpotBugs (Java), Pylint (Python)
  • • Stress testing: Apache JMeter, Gatling, Locust

Common Mistakes

❌ Using sleep() as synchronization

Adding arbitrary delays doesn't fix races - it just makes them harder to reproduce. Use proper locks.

// ❌ WRONG: Sleep is not synchronization!
if (!fileExists(file)) {
    time.sleep(0.1);  // Attacker still wins this race
    createFile(file);
}

❌ Double-checked locking without volatile

Classic broken singleton pattern - requires proper memory barriers.

// ❌ BROKEN: Double-checked locking without volatile
private static Singleton instance;

public static Singleton getInstance() {
    if (instance == null) {  // Check 1 (unsynchronized)
        synchronized (Singleton.class) {
            if (instance == null) {  // Check 2 (synchronized)
                instance = new Singleton();  // Can be partially constructed!
            }
        }
    }
    return instance;
}

// ✅ CORRECT: Add volatile keyword
private static volatile Singleton instance;

❌ Locking at wrong granularity

Lock too much = performance issues. Lock too little = race conditions.

// ❌ TOO COARSE: Entire method locked
public synchronized void processLargeFile() {
    readFile();      // Lock held during I/O - bad!
    processData();   // Lock held during computation - bad!
    writeResults();  // Lock held during more I/O - bad!
}

// ✅ CORRECT: Lock only critical section
public void processLargeFile() {
    byte[] data = readFile();      // No lock
    byte[] results = processData(data);  // No lock
    
    synchronized (this) {
        writeResults(results);     // Lock only during shared state modification
    }
}

Framework Solutions

Django (Python)

from django.db import transaction
from django.db.models import F

# ✅ Atomic update
Product.objects.filter(
    id=product_id
).update(
    quantity=F('quantity') - 1
)

# ✅ Select for update
with transaction.atomic():
    product = Product.objects.select_for_update().get(id=product_id)
    if product.quantity > 0:
        product.quantity -= 1
        product.save()

Rails (Ruby)

# ✅ Optimistic locking
class Product < ApplicationRecord
  self.locking_column = :lock_version
end

product.update!(quantity: product.quantity - 1)
# Raises StaleObjectError on concurrent modification

# ✅ Pessimistic locking
Product.transaction do
  product = Product.lock.find(product_id)
  product.quantity -= 1
  product.save!
end

Race Condition Labs

Practice exploiting and preventing race condition vulnerabilities.

Race Conditions PortSwigger hard
Limit overrunMulti-endpoint racesSingle-endpoint races

Evidence to Capture

The harness: The exact script or tool configuration used to fire the requests, the count, and how they were synchronised. A loop of sequential curl calls cannot produce a race, so the harness is part of the finding.

Outcome difference: The end state read back with a fresh request showing more operations succeeded than the invariant allows: the balance, the redemption count, the duplicate rows, the number of orders.

Database or file view: The two interleaved statements from the query log, or the duplicate rows with their timestamps, which shows the window rather than inferring it.

Window size: The measured gap between check and use, and how many parallel requests fit inside it. This tells the developer which lock or constraint would close it.

State-machine case: For order-dependent flows, the accepted out-of-order transitions, listed in the sequence they were sent, with the resulting status.

Likely False Positives

  • Sequential requests treated as concurrent: If the second request fails because the first already committed, the invariant held. A race requires released-at-once requests, not a fast loop.
  • Retry duplicates: A client retry after a timeout creates a second row that looks like double processing. Check request ids, idempotency keys and timestamps before reporting.
  • Rate-limit rejections read as success: A 429 means the control engaged. Only read-back state that violates the invariant counts.
  • Single-replica test declared fixed: An in-process lock passes on one instance and fails behind a load balancer. If production runs several replicas, the test must too.
  • Timeout counted as a successful race: A request that never returned proves nothing. Confirm the state change by reading the resource afterwards.

Testing Checklist

Discovery

  • ☐ List the check-then-act pairs: balance or credit spend, coupon and gift-card redemption, vote and like counters, invite acceptance, inventory reservation, OTP verification.
  • ☐ For each, establish where the guard lives — application code, a database constraint, or a queue — because that determines the fix.
  • ☐ Note the deployment topology: replicas, workers and cron jobs change which locking primitive is valid.

Confirmation

  • ☐ Release N parallel requests at one resource with a barrier, then read the resource back and count the effects.
  • ☐ Repeat across replicas or through the load balancer, with the requests spread over connections rather than pooled on one.
  • ☐ For state machines, send transitions out of order and in duplicate, and record which are accepted.

Impact

  • ☐ Quantify the loss: funds transferred, inventory oversold, redemptions beyond the limit, privilege granted twice. State whether the effect repeats at will.
  • ☐ Check whether the race is exploitable once per resource, or indefinitely, since only the second is a business-logic failure of consequence.
  • ☐ Confirm whether the same window exists on the compensating control itself, such as the duplicate check in front of the payment call.

Remediation

Structural fix

  • • Make the invariant the database enforcement: a conditional update that succeeds only when the precondition still holds, or a unique constraint on the resource and actor pair.
  • • Take the lock with the read: SELECT ... FOR UPDATE inside one transaction, or a state transition expressed as UPDATE ... WHERE status = 'pending' and checking the affected row count.
  • • Replace check-then-act on the filesystem with atomic operations: exclusive create, open followed by a descriptor read, and no path re-resolution between the two.

Compensating control

  • • Require an idempotency key on every state-changing request and store the outcome, so a replay returns the first result instead of applying a second.
  • • Use a distributed lock with fencing tokens where the datastore cannot express the invariant, and treat the lock as an optimisation rather than the guarantee.
  • • Serialise per-resource work through a queue when throughput allows, which removes the concurrent path entirely.

CWE References: CWE-362