Skip to main content
← Back to list
01Issue
FeatureClosedSwamp Club
AssigneesNone

Datastore lock: allow concurrent reads across different models

Opened by swampadmin · 9/22/2025

Problem

Currently the datastore uses a single global file lock. Running swamp model method run on one model blocks all other model method runs, even on completely unrelated models. This makes it impossible to run operations on different models in parallel (e.g. querying prod-vms while also running a workflow on a different model).

Context

When running fleet operations across 100+ VMs, each method call can take 30-90 seconds. Being unable to run a quick query on a different model while a long operation is in progress feels unnecessarily rigid.

Proposed Solution

Consider a more granular locking strategy:

  • Per-model locks — each model instance gets its own lock, so unrelated models don't block each other
  • Read/write locks — read-only operations (like get, list) could run concurrently, with exclusive locks only for writes
  • Per-method locks — even within a model, independent methods on different resources could potentially run concurrently

The simplest improvement would be per-model locking, so that running a method on model A doesn't block an unrelated method on model B.

Current behavior

# Terminal 1: starts a 90-second healthCheck on prod-vms
swamp model method run prod-vms healthCheck --json

# Terminal 2: tries to check a different model — blocked until Terminal 1 finishes
swamp model method run dev-vms checkService --input service=docker --json
# ^ waits for datastore lock, eventually times out

Expected behavior

Operations on different model instances should not block each other.

02Bog Flow
OPENTRIAGEDIN PROGRESSCLOSED

Closed

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.