Race Conditions Remediation
Race conditions occur when multiple threads or processes access shared resources concurrently without proper synchronization, leading to inconsistent state, privilege escalation, or data corruption.
Critical Impact
Race Condition Attack Flow
Understanding Race Conditions
Common Race Condition Types
- • TOCTOU: Time-of-Check-Time-of-Use
- • Database races: Concurrent transactions
- • File system races: Symlink attacks
- • State transitions: Order-dependent operations
- • Double-spend: Balance check race
Attack Scenario
- Thread 1 checks if file exists
- Thread 2 creates malicious symlink
- Thread 1 opens file (now follows symlink)
- Attacker reads/writes sensitive file
TOCTOU Prevention
Best Practice
Vulnerable File Access (TOCTOU)
// ❌ VULNERABLE: Check-then-act pattern
if (fs.existsSync(filename)) {
// Race window here! File could be deleted or replaced
const content = fs.readFileSync(filename);
processFile(content);
}
// ✅ SECURE: Try-and-catch pattern
try {
const content = fs.readFileSync(filename);
processFile(content);
} catch (err) {
// Handle missing file
}
// ✅ SECURE: Atomic file operations (Node.js)
const fd = fs.openSync(filename, 'r', 0o666);
try {
const content = fs.readFileSync(fd);
processFile(content);
} finally {
fs.closeSync(fd);
}Python Safe File Operations
# ❌ VULNERABLE: Separate check and open
import os
if os.path.exists(filename):
with open(filename) as f: # Race window!
data = f.read()
# ✅ SECURE: Try to open directly
try:
with open(filename, 'r') as f:
data = f.read()
except FileNotFoundError:
# Handle missing file
pass
# ✅ SECURE: Use exclusive creation flag
import os
try:
# O_CREAT | O_EXCL ensures atomic creation
fd = os.open(filename, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(fd, 'w') as f:
f.write(sensitive_data)
except FileExistsError:
# File already exists - safe failure
passDatabase Race Conditions
Vulnerable Balance Check
-- ❌ VULNERABLE: Separate read and update
-- Thread 1: SELECT balance = 100
-- Thread 2: SELECT balance = 100
-- Thread 1: UPDATE balance = 100 - 60 = 40
-- Thread 2: UPDATE balance = 100 - 50 = 50 ← Wrong! Should be -10
-- Query 1: Check balance
SELECT balance FROM accounts WHERE user_id = 123;
-- Race window here!
-- Query 2: Deduct money
UPDATE accounts
SET balance = balance - 50
WHERE user_id = 123;Secure: Atomic Operations
-- ✅ SECURE: Single atomic transaction with constraint
UPDATE accounts
SET balance = balance - 50
WHERE user_id = 123
AND balance >= 50; -- Check happens atomically
-- Check affected rows
IF @@ROWCOUNT = 0 THEN
RAISE 'Insufficient funds';
END IF;
-- ✅ SECURE: Use database transactions with serializable isolation
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
SELECT balance FROM accounts WHERE user_id = 123 FOR UPDATE;
-- Row is now locked, no other transaction can modify it
IF balance >= 50 THEN
UPDATE accounts SET balance = balance - 50 WHERE user_id = 123;
ELSE
ROLLBACK;
RAISE 'Insufficient funds';
END IF;
COMMIT;Application Synchronization
Python Threading Locks
import threading
# ❌ VULNERABLE: No synchronization
balance = 1000
def withdraw(amount):
global balance
if balance >= amount:
# Race window!
balance -= amount
return True
return False
# ✅ SECURE: Use lock for critical section
balance = 1000
balance_lock = threading.Lock()
def withdraw_safe(amount):
global balance
with balance_lock:
if balance >= amount:
balance -= amount
return True
return False
# ✅ SECURE: Use thread-safe data structures
from queue import Queue
from threading import Semaphore
# Semaphore for resource limiting
max_connections = Semaphore(10)
def process_request():
with max_connections:
# Only 10 threads can be here at once
handle_connection()Java Synchronized Methods
// ❌ VULNERABLE: Non-atomic check-then-act
private int balance = 1000;
public boolean withdraw(int amount) {
if (balance >= amount) { // Race!
balance -= amount;
return true;
}
return false;
}
// ✅ SECURE: Synchronized method
private int balance = 1000;
public synchronized boolean withdraw(int amount) {
if (balance >= amount) {
balance -= amount;
return true;
}
return false;
}
// ✅ SECURE: Use concurrent collections
import java.util.concurrent.atomic.AtomicInteger;
private AtomicInteger balance = new AtomicInteger(1000);
public boolean withdraw(int amount) {
while (true) {
int current = balance.get();
if (current < amount) {
return false;
}
int next = current - amount;
if (balance.compareAndSet(current, next)) {
return true; // Success!
}
// CAS failed, retry
}
}
// ✅ SECURE: ReentrantLock with conditions
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
private final Lock lock = new ReentrantLock();
public boolean withdraw(int amount) {
lock.lock();
try {
if (balance >= amount) {
balance -= amount;
return true;
}
return false;
} finally {
lock.unlock();
}
}Distributed System Races
Redis Distributed Locks
// ✅ SECURE: Redis distributed lock with Redlock algorithm
const redis = require('redis');
const { promisify } = require('util');
async function acquireLock(client, lockKey, ttl = 5000) {
const lockValue = crypto.randomUUID();
// SET with NX (only if not exists) and PX (TTL in ms)
const result = await client.set(
lockKey,
lockValue,
'NX',
'PX',
ttl
);
if (result === 'OK') {
return lockValue; // Lock acquired
}
return null; // Lock held by another process
}
async function releaseLock(client, lockKey, lockValue) {
// Use Lua script for atomic check-and-delete
const script = `
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
else
return 0
end
`;
await client.eval(script, 1, lockKey, lockValue);
}
// Usage
async function processWithLock() {
const lockValue = await acquireLock(redisClient, 'resource:123');
if (!lockValue) {
throw new Error('Could not acquire lock');
}
try {
// Critical section
await performTransaction();
} finally {
await releaseLock(redisClient, 'resource:123', lockValue);
}
}Database-Level Pessimistic Locking
-- ✅ SECURE: Row-level locking with SELECT FOR UPDATE
BEGIN;
-- Lock the row exclusively
SELECT * FROM inventory
WHERE product_id = 123
FOR UPDATE;
-- No other transaction can modify this row until we commit
UPDATE inventory
SET quantity = quantity - 1
WHERE product_id = 123;
COMMIT;
-- ✅ SECURE: Named advisory locks (PostgreSQL)
SELECT pg_advisory_lock(123456);
-- Critical section - only one session can hold this lock
SELECT pg_advisory_unlock(123456);
-- ✅ SECURE: Optimistic locking with version column
UPDATE products
SET
quantity = quantity - 1,
version = version + 1
WHERE
product_id = 123
AND version = @expected_version;
-- Check affected rows
IF @@ROWCOUNT = 0 THEN
RAISE 'Concurrent modification detected';
END IF;State Transition Safety
// ❌ VULNERABLE: Unprotected state transitions
class Order {
constructor() {
this.status = 'pending';
}
approve() {
this.status = 'approved'; // Race!
}
cancel() {
this.status = 'cancelled'; // Could overwrite approval!
}
}
// ✅ SECURE: Atomic state transitions with validation
class OrderSafe {
constructor() {
this.status = 'pending';
this.version = 0;
}
async transitionTo(newStatus, expectedVersion) {
// Database enforces version check atomically
const result = await db.query(
`UPDATE orders
SET status = $1, version = version + 1
WHERE id = $2
AND status = $3
AND version = $4`,
[newStatus, this.id, this.status, expectedVersion]
);
if (result.rowCount === 0) {
throw new Error('Invalid state transition or concurrent modification');
}
this.status = newStatus;
this.version++;
}
async approve() {
if (this.status !== 'pending') {
throw new Error('Can only approve pending orders');
}
await this.transitionTo('approved', this.version);
}
}
// ✅ SECURE: State machine with explicit transitions
const validTransitions = {
'pending': ['approved', 'cancelled'],
'approved': ['shipped'],
'shipped': ['delivered'],
'cancelled': [], // Terminal state
'delivered': [] // Terminal state
};
function canTransition(currentState, newState) {
return validTransitions[currentState]?.includes(newState) || false;
}Testing Verification
Testing Challenge
Concurrent Testing (Python)
import threading
import time
# Test for race conditions with concurrent threads
def test_concurrent_withdrawal():
balance = Account(1000)
threads = []
# Try to withdraw $600 twice simultaneously (should fail once)
def withdraw_thread():
result = balance.withdraw(600)
print(f"Withdrawal {'succeeded' if result else 'failed'}")
# Launch multiple threads
for _ in range(2):
t = threading.Thread(target=withdraw_thread)
threads.append(t)
t.start()
# Wait for completion
for t in threads:
t.join()
final_balance = balance.get_balance()
print(f"Final balance: {final_balance}")
# Should be either 400 (one succeeded) or 1000 (both failed)
# Should NEVER be -200 (both succeeded - race condition!)
assert final_balance in [400, 1000], "Race condition detected!"
# Stress test with many threads
def stress_test_race_condition():
account = Account(10000)
threads = []
def worker():
for _ in range(100):
account.withdraw(1)
time.sleep(0.0001) # Increase race window
# 10 threads, 100 withdrawals each = 1000 total
for _ in range(10):
t = threading.Thread(target=worker)
threads.append(t)
t.start()
for t in threads:
t.join()
expected_balance = 10000 - 1000
actual_balance = account.get_balance()
print(f"Expected: {expected_balance}, Actual: {actual_balance}")
assert actual_balance == expected_balance, "Race condition detected!"Testing Tools
Race Condition Detection Tools
- • ThreadSanitizer (TSan): Detects data races in C/C++/Go
- • Java Concurrency Stress Tests: jcstress framework
- • Python threading analysis: pytest-timeout, pytest-xdist
- • Static analysis: FindBugs, SpotBugs (Java), Pylint (Python)
- • Stress testing: Apache JMeter, Gatling, Locust
Common Mistakes
❌ Using sleep() as synchronization
Adding arbitrary delays doesn't fix races - it just makes them harder to reproduce. Use proper locks.
// ❌ WRONG: Sleep is not synchronization!
if (!fileExists(file)) {
time.sleep(0.1); // Attacker still wins this race
createFile(file);
}❌ Double-checked locking without volatile
Classic broken singleton pattern - requires proper memory barriers.
// ❌ BROKEN: Double-checked locking without volatile
private static Singleton instance;
public static Singleton getInstance() {
if (instance == null) { // Check 1 (unsynchronized)
synchronized (Singleton.class) {
if (instance == null) { // Check 2 (synchronized)
instance = new Singleton(); // Can be partially constructed!
}
}
}
return instance;
}
// ✅ CORRECT: Add volatile keyword
private static volatile Singleton instance;❌ Locking at wrong granularity
Lock too much = performance issues. Lock too little = race conditions.
// ❌ TOO COARSE: Entire method locked
public synchronized void processLargeFile() {
readFile(); // Lock held during I/O - bad!
processData(); // Lock held during computation - bad!
writeResults(); // Lock held during more I/O - bad!
}
// ✅ CORRECT: Lock only critical section
public void processLargeFile() {
byte[] data = readFile(); // No lock
byte[] results = processData(data); // No lock
synchronized (this) {
writeResults(results); // Lock only during shared state modification
}
}Framework Solutions
Django (Python)
from django.db import transaction
from django.db.models import F
# ✅ Atomic update
Product.objects.filter(
id=product_id
).update(
quantity=F('quantity') - 1
)
# ✅ Select for update
with transaction.atomic():
product = Product.objects.select_for_update().get(id=product_id)
if product.quantity > 0:
product.quantity -= 1
product.save()Rails (Ruby)
# ✅ Optimistic locking
class Product < ApplicationRecord
self.locking_column = :lock_version
end
product.update!(quantity: product.quantity - 1)
# Raises StaleObjectError on concurrent modification
# ✅ Pessimistic locking
Product.transaction do
product = Product.lock.find(product_id)
product.quantity -= 1
product.save!
endRace Condition Labs
Practice exploiting and preventing race condition vulnerabilities.
Evidence to Capture
The harness: The exact script or tool configuration used to fire the requests, the count, and how they were synchronised. A loop of sequential curl calls cannot produce a race, so the harness is part of the finding.
Outcome difference: The end state read back with a fresh request showing more operations succeeded than the invariant allows: the balance, the redemption count, the duplicate rows, the number of orders.
Database or file view: The two interleaved statements from the query log, or the duplicate rows with their timestamps, which shows the window rather than inferring it.
Window size: The measured gap between check and use, and how many parallel requests fit inside it. This tells the developer which lock or constraint would close it.
State-machine case: For order-dependent flows, the accepted out-of-order transitions, listed in the sequence they were sent, with the resulting status.
Likely False Positives
- Sequential requests treated as concurrent: If the second request fails because the first already committed, the invariant held. A race requires released-at-once requests, not a fast loop.
- Retry duplicates: A client retry after a timeout creates a second row that looks like double processing. Check request ids, idempotency keys and timestamps before reporting.
- Rate-limit rejections read as success: A 429 means the control engaged. Only read-back state that violates the invariant counts.
- Single-replica test declared fixed: An in-process lock passes on one instance and fails behind a load balancer. If production runs several replicas, the test must too.
- Timeout counted as a successful race: A request that never returned proves nothing. Confirm the state change by reading the resource afterwards.
Testing Checklist
Discovery
- ☐ List the check-then-act pairs: balance or credit spend, coupon and gift-card redemption, vote and like counters, invite acceptance, inventory reservation, OTP verification.
- ☐ For each, establish where the guard lives — application code, a database constraint, or a queue — because that determines the fix.
- ☐ Note the deployment topology: replicas, workers and cron jobs change which locking primitive is valid.
Confirmation
- ☐ Release N parallel requests at one resource with a barrier, then read the resource back and count the effects.
- ☐ Repeat across replicas or through the load balancer, with the requests spread over connections rather than pooled on one.
- ☐ For state machines, send transitions out of order and in duplicate, and record which are accepted.
Impact
- ☐ Quantify the loss: funds transferred, inventory oversold, redemptions beyond the limit, privilege granted twice. State whether the effect repeats at will.
- ☐ Check whether the race is exploitable once per resource, or indefinitely, since only the second is a business-logic failure of consequence.
- ☐ Confirm whether the same window exists on the compensating control itself, such as the duplicate check in front of the payment call.
Remediation
Structural fix
- • Make the invariant the database enforcement: a conditional update that succeeds only when the precondition still holds, or a unique constraint on the resource and actor pair.
- • Take the lock with the read:
SELECT ... FOR UPDATEinside one transaction, or a state transition expressed asUPDATE ... WHERE status = 'pending'and checking the affected row count. - • Replace check-then-act on the filesystem with atomic operations: exclusive create, open followed by a descriptor read, and no path re-resolution between the two.
Compensating control
- • Require an idempotency key on every state-changing request and store the outcome, so a replay returns the first result instead of applying a second.
- • Use a distributed lock with fencing tokens where the datastore cannot express the invariant, and treat the lock as an optimisation rather than the guarantee.
- • Serialise per-resource work through a queue when throughput allows, which removes the concurrent path entirely.
CWE References: CWE-362