Sync & CRDT System

Sync Overview

haex-vault uses a CRDT (Conflict-free Replicated Data Type) based sync system that enables seamless offline-first operation with automatic conflict resolution.

CRDT Fundamentals

CRDTs are data structures that can be replicated across multiple computers, modified independently, and merged without conflicts.

CRDT Columns

Every synced table has three special columns that enable CRDT functionality:

-- All synced tables have these columns
CREATE TABLE haex_passwords (
  id TEXT PRIMARY KEY,
  title TEXT NOT NULL,
  username TEXT,
  password TEXT,

  -- CRDT Columns (added automatically):
  haex_timestamp TEXT NOT NULL,      -- Max HLC across all columns
  haex_column_hlcs TEXT NOT NULL     -- JSON object with HLC per column
                     DEFAULT '{}',
  haex_tombstone INTEGER NOT NULL    -- 0 = active, 1 = soft-deleted
                     DEFAULT 0
);

Hybrid Logical Clocks

HLCs combine physical time with logical counters to create globally unique, monotonically increasing timestamps without requiring synchronized clocks.

HLC Format: ISO 8601 timestamp + Device ID suffix

"2024-01-03T10:45:32.123-A"
         │           │     │
    ISO 8601    milliseconds  Device ID

Key benefits of HLCs:

• Lexicographically sortable - can use simple string comparison
• Globally unique - device ID suffix prevents collisions
• Causally ordered - respects happens-before relationships

Column-Level Sync

Unlike row-level sync, haex-vault tracks changes at the column level. This means concurrent edits to different columns of the same row are automatically merged without data loss.

Device A: UPDATE passwords SET title = 'Gmail' WHERE id = 'abc'
         (timestamp: 2024-01-03T10:00:00-A)

Device B: UPDATE passwords SET password = 'newpass' WHERE id = 'abc'
         (timestamp: 2024-01-03T10:00:15-B)

Result after sync:
- title = 'Gmail'     (from Device A)
- password = 'newpass' (from Device B)

Both changes are preserved!

Column HLC Storage

// haex_column_hlcs JSON structure
{
  "id": "2024-01-03T10:45:00.000-A",
  "title": "2024-01-03T10:45:15.500-A",
  "username": "2024-01-03T10:45:20.100-B",
  "password": "2024-01-03T10:45:30.200-B"
}

// haex_timestamp = max(all column HLCs)
// = "2024-01-03T10:45:30.200-B"

Sync Flow

The sync process involves three main operations: push, pull, and realtime subscriptions.

┌─────────────────────────────────────────────────────────────┐
│                    USER MAKES CHANGE                         │
│              (Insert/Update/Delete Row)                      │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│   SQLite Trigger fires                                       │
│   → Entry added to haex_crdt_dirty_tables                   │
│   → Rust emits 'crdt:dirty-tables-changed' event            │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│   Sync Orchestrator (debounced 300ms)                        │
│   → Scans dirty tables for changes since last push          │
│   → Generates column-level changes                          │
│   → Encrypts each value with vault key                      │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│   POST /sync/push → haex-sync-server                         │
│   → UPSERT into sync_changes (partitioned by vault_id)      │
│   → HLC comparison: only store if new HLC > existing        │
│   → Trigger Supabase Realtime notification                  │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│   Other devices receive Realtime event                       │
│   → Trigger debounced pull                                  │
│   → GET /sync/pull?afterUpdatedAt=lastPullTimestamp         │
│   → Decrypt values, apply with HLC comparison               │
│   → Emit 'haextension:sync:tables-updated' event            │
└─────────────────────────────────────────────────────────────┘

Push Operation

When you make changes locally, the sync orchestrator detects dirty tables, scans for changes since the last push, encrypts each value, and sends them to the server in batches.

// Column Change Structure (sent to server)
{
  tableName: "haex_passwords",
  rowPks: '{"id": "abc-123"}',
  columnName: "password",
  hlcTimestamp: "2024-01-03T10:45:30.200-A",
  encryptedValue: "base64...",  // AES-256-GCM
  nonce: "base64...",
  batchId: "uuid",
  batchSeq: 1,
  batchTotal: 5
}

Pull Operation

Pull fetches changes from the server since the last pull timestamp. Each change is decrypted and compared against local data using HLC timestamps. Only newer changes are applied.

// Pull Logic (simplified)
for (const change of serverChanges) {
  const localRow = await getRow(change.tableName, change.rowPks)
  const localHlc = localRow?.haex_column_hlcs[change.columnName]

  if (!localHlc || change.hlcTimestamp > localHlc) {
    // Remote wins - apply the change
    const decryptedValue = decrypt(change.encryptedValue, vaultKey)
    await updateColumn(change.tableName, change.rowPks,
                       change.columnName, decryptedValue)
    await updateColumnHlc(change.tableName, change.rowPks,
                          change.columnName, change.hlcTimestamp)
  }
  // else: Local wins - keep existing value
}

Realtime Updates

haex-vault subscribes to Supabase Realtime channels to receive instant notifications when other devices push changes. This triggers a debounced pull operation.

// Supabase Realtime Subscription
const channel = supabase
  .channel(`sync_changes:${vaultId}`)
  .on('postgres_changes', {
    event: 'INSERT',
    schema: 'public',
    table: `sync_changes_${vaultId.replace(/-/g, '_')}`  // Partition
  }, (payload) => {
    // Skip if change is from this device
    if (payload.new.device_id === myDeviceId) return

    // Trigger debounced pull (500ms)
    triggerPull()
  })
  .subscribe()

Soft Delete with Tombstones

haex-vault uses soft deletion instead of actual DELETE statements. This ensures deletes can be properly synchronized across devices.

-- Instead of:
DELETE FROM haex_passwords WHERE id = 'abc-123';

-- Use:
UPDATE haex_passwords SET haex_tombstone = 1 WHERE id = 'abc-123';

-- Query active records:
SELECT * FROM haex_passwords WHERE haex_tombstone = 0;

Soft deletion allows the delete operation to be treated like any other column change, syncing properly through the CRDT system. The row remains in the database with haex_tombstone = 1.

Sync Events

After a successful pull, events are emitted automatically by the sync orchestrator to notify the application that data has changed. Extensions don't need to emit these events - they only need to listen and react to them.

Permission-Based Event Filtering

For security and privacy, sync events are filtered based on each extension's database permissions. Extensions only receive notifications about tables they have access to.

When sync completes, haex-vault checks each extension's DB permissions and only includes table names that match:

• Wildcard permission (*) - sees all tables
• Prefix match (e.g., publickey__extname__*) - sees tables with that prefix
• Exact match - sees only that specific table

Handling Sync Events in Extensions

The event payload includes a list of table names that were updated (filtered to only tables your extension can access). Check if any of your tables are in the list and reload data accordingly.

// Listen for sync events in your extension
vault.on('sync:tables-updated', async (event) => {
  const tables = event.data?.tables || []

  // Check if your tables were updated
  if (tables.some(t => t.includes('my_extension_table'))) {
    // Reload your data
    await reloadDataAsync()
  }
})