Skip to main content
← Back to list
01Issue
BugClosedSwamp Club
AssigneesNone

Race condition: repo index concurrent with vault put crashes with ENOTEMPTY

Opened by swampadmin · 10/28/2024

Summary

Running swamp repo index concurrently with swamp vault put crashes with ENOTEMPTY when both processes attempt to rebuild the secrets symlink index simultaneously.

This is a separate trigger from #487 (which covered concurrent vault put only). The repo index path goes through rebuildAllindexVaultrefreshSecretsIndex, colliding with vault put's handleVaultSecretUpdatedindexVaultrefreshSecretsIndex.

Reproduction

swamp init
swamp vault create local_encryption test-vault
swamp vault put test-vault SEED=x

# Run concurrently:
for i in 1 2 3 4 5; do
  swamp vault put test-vault "KEY_${i}=val${i}" &
  swamp repo index --json &
done
wait

The failure is intermittent but triggers frequently (first attempt in UAT).

Error

FTL error Error: Directory not empty (os error 66): remove '.../vaults/test-vault/secrets'
    at async Object.remove (ext:deno_fs/30_fs.js:258:3)
    at async SymlinkRepoIndexService.refreshSecretsIndex (symlink_repo_index_service.ts:786:9)
    at async SymlinkRepoIndexService.indexVault (symlink_repo_index_service.ts:765:7)
    at async SymlinkRepoIndexService.rebuildAll (symlink_repo_index_service.ts:360:9)
    at async Command.<anonymous> (repo_index.ts:86:22)

Stack Traces

Two code paths collide on refreshSecretsIndex:

vault put path:

handleVaultSecretUpdated → indexVault → refreshSecretsIndex

repo index path:

rebuildAll → indexVault → refreshSecretsIndex

Root Cause

refreshSecretsIndex performs a non-atomic remove + recreate of the secrets symlink directory. When repo index rebuilds the full index while vault put triggers its own index update, both enter refreshSecretsIndex and race on the directory contents.

Found By

Adversarial concurrency UAT in swamp-uattests/adversarial/concurrency_test.ts, test E ("swamp repo index concurrent with vault put does not crash or corrupt the index").

02Bog Flow
OPENTRIAGEDIN PROGRESSCLOSED

Closed

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.