Skip to main content
← Back to list
01Issue
FeatureClosedSwamp Club
AssigneesNone

Race condition in concurrent vault put: refreshSecretsIndex ENOTEMPTY

Opened by swampadmin · 10/25/2024

Summary

Concurrent vault put operations can fail with ENOTEMPTY when SymlinkRepoIndexService.refreshSecretsIndex performs a non-atomic remove + recreate of the secrets directory.

Reproduction

Run 5 concurrent vault put processes writing different keys to the same vault:

swamp init
swamp vault create local_encryption test-vault

# Run these concurrently (not sequentially):
swamp vault put test-vault KEY_1=val1 &
swamp vault put test-vault KEY_2=val2 &
swamp vault put test-vault KEY_3=val3 &
swamp vault put test-vault KEY_4=val4 &
swamp vault put test-vault KEY_5=val5 &
wait

The failure is intermittent — it depends on process scheduling timing.

Error

FTL error Error: Directory not empty (os error 66): remove '.../vaults/test-vault/secrets'
    at async Object.remove (ext:deno_fs/30_fs.js:258:3)
    at async SymlinkRepoIndexService.refreshSecretsIndex (symlink_repo_index_service.ts:786:9)
    at async SymlinkRepoIndexService.indexVault (symlink_repo_index_service.ts:765:7)
    at async SymlinkRepoIndexService.handleVaultSecretUpdated (symlink_repo_index_service.ts:215:5)

Root Cause

refreshSecretsIndex removes the secrets directory and recreates it. When two processes run concurrently:

  1. Process A removes the directory
  2. Process B creates new files in the directory (or vice versa)
  3. Process B's remove fails with ENOTEMPTY because Process A already wrote into it

This is a TOCTOU (time-of-check-time-of-use) race on the directory contents.

Suggested Fix

Replace the remove + recreate pattern with an atomic approach, e.g.:

  • Build the new index in a temp directory, then atomic rename
  • Or use a retry loop around the remove + recreate

Found By

Adversarial concurrency UAT in swamp-uattests/adversarial/concurrency_test.ts, test B ("swamp vault put to different keys concurrently preserves all keys").

02Bog Flow
OPENTRIAGEDIN PROGRESSCLOSED

Closed

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.