Dev #6

Merged
sandrodach merged 6 commits from dev into main 2026-10-26 18:54:23 +01:00
13 changed files with 1028 additions and 85 deletions
Showing only changes of commit ef8cf8f022 - Show all commits

No files matched your search

+35 -25
View File
@@ -64,43 +64,53 @@ Alle WebSocket-Nachrichten werden als JSON mit einheitlicher Hülle übertragen:
---
## 🔒 4. Dokument-Sperrsystem (Document Locking)
---
Um Schreibkonflikte bei Notizen im Workspace zu vermeiden, verwendet MindSift ein **serverseitiges Lease-Locking mit Heartbeat**:
## ⚡ 4. CRDT-Echtzeitprotokoll (Yjs Collaborative Editing)
Seit Version 2.1 unterstützt MindSift **echte, konfliktfreie Mehrbenutzer-Kollaboration** für Notizen via Yjs CRDT. Clients tauschen binäre Delta-Updates und Live-Awareness-Informationen über dedizierte WebSocket-Nachrichten aus:
### CRDT-Nachrichtentypen
| Event | Richtung | Zweck / Payload |
|---|---|---|
| `CRDT_JOIN` | Client ➔ Server | Tritt dem Dokumentenraum (`doc:{docId}`) bei. |
| `CRDT_INIT` | Server ➔ Client | Sendet den aktuellen Basiszustand (`stateBlob`) und offene Delta-Updates (`updates[]`) an den Client. |
| `CRDT_UPDATE` | Bidirektional | Überträgt inkrementelle binäre Yjs-Updates (Base64-kodiert). Server speichert in `document_crdt_updates` und verteilt an alle anderen Raumnutzer. |
| `CRDT_COMPACT` | Client ➔ Server | Komprimiert mehrere Updates zu einem konsolidierten Snapshot in `document_crdt_state`. |
| `CRDT_AWARENESS` | Bidirektional | Überträgt Live-Präsenz (Fokus auf `blockId`, Benutzername, Farbe). |
| `CRDT_AWARENESS_LEAVE` | Bidirektional | Signalisiert, dass ein Nutzer den Fokus verloren hat oder das Dokument verlassen hat. |
| `CRDT_LEAVE` | Client ➔ Server | Verlässt den Dokumentenraum explizit. |
```mermaid
sequenceDiagram
autonumber
actor UserA as Benutzer A
participant WS as WebSocket Hub
participant WS as WebSocket Hub & SQLite
actor UserB as Benutzer B
UserA->>WS: LOCK_DOC (docId: "doc_123")
WS-->>UserA: Lock bestätigt (30s TTL)
WS->>UserB: DOC_LOCKED (docId: "doc_123", lockedBy: "Benutzer A")
Note over UserB: Editor gesperrt (Schreibschutz aktiv)
UserA->>WS: CRDT_JOIN (docId: "doc_123")
WS-->>UserA: CRDT_INIT (stateBlob, updates)
UserB->>WS: CRDT_JOIN (docId: "doc_123")
WS-->>UserB: CRDT_INIT (stateBlob, updates)
loop Alle 15 Sekunden
UserA->>WS: HEARTBEAT_DOC (docId: "doc_123")
WS-->>UserA: TTL um 30s verlängert
end
UserA->>WS: CRDT_UPDATE (docId, updateBlob)
WS->>WS: Append in document_crdt_updates
WS->>UserB: CRDT_UPDATE (docId, updateBlob)
Note over UserB: Yjs wendet Delta atomar an (0 Konflikte)
alt Reguläres Schließen
UserA->>WS: UNLOCK_DOC (docId: "doc_123")
WS->>UserB: DOC_UNLOCKED (docId: "doc_123")
Note over UserB: Editor wieder freigegeben
else Verbindungsabbruch / Tab geschlossen
Note over WS: Heartbeat bleibt aus (TTL läuft ab)
WS->>UserB: DOC_UNLOCKED (Automatisch nach Ablauf)
end
UserA->>WS: CRDT_AWARENESS (docId, focusedBlockId)
WS->>UserB: CRDT_AWARENESS (docId, user, focusedBlockId)
Note over UserB: Avatar-Badge erscheint am Block
```
### Locking-Events
- **`LOCK_DOC` (Client ➔ Server):** Fordert eine 30-Sekunden-Sperre für ein Dokument an.
- **`HEARTBEAT_DOC` (Client ➔ Server):** Verlängert die Sperre alle 15 Sekunden, solange der Benutzer das Dokument geöffnet hat.
- **`UNLOCK_DOC` (Client ➔ Server):** Gibt die Sperre beim Verlassen des Dokuments frei.
- **`DOC_LOCKED` (Server ➔ Clients):** Informiert andere Benutzer, dass das Dokument schreibgeschützt ist.
- **`DOC_UNLOCKED` (Server ➔ Clients):** Hebt den Schreibschutz auf.
---
## 🔒 5. Abwärtskompatibles Document-Lease-Locking
Vor Version 2.1 nutzte MindSift ein serverseitiges Lease-Locking mit 30-Sekunden-Heartbeats. Durch die CRDT-Engine wurde die blockierende Bearbeitungssperre (`isLockedByOther`) abgelöst:
- **Aktuelles Verhalten:** Mehrere Benutzer mit Bearbeitungsrechten können zeitgleich dasselbe Dokument bearbeiten.
- **Presence & Locks:** Die Events `LOCK_DOC` und `HEARTBEAT_DOC` bleiben abwärtskompatibel erhalten und dienen nun der reinen Präsenzanzeige (z.B. gelber Statusindikator „wird gerade von X angesehen").
- **Schreibschutz:** Ein echter Schreibschutz (`isReadOnly`) greift ausschließlich noch dann, wenn der Nutzer nur über Leserechte (`permission === 'read'`) für die Notiz oder den Ordner verfügt.
---
+11 -8
View File
@@ -8,7 +8,7 @@ Diese Datei dokumentiert eine ehrliche und schonungslose technische Analyse alle
| ID | Bereich | Problem / Schwachstelle | Schweregrad | Risiko im Produktivbetrieb |
|---|---|---|---|---|
| **FLAW-01** | Workspace Sync | Kein echtes CRDT/OT – Lock-Heartbeat-Modell mit Last-Write-Wins | **Kritisch** | Zeilenverlust bei gleichzeitigen Bearbeitungen oder Netzschwankungen |
| **FLAW-01** | Workspace Sync | Kein echtes CRDT/OT – Lock-Heartbeat-Modell mit Last-Write-Wins | **Gelöst (v2.1)** | Behoben durch Yjs CRDT-Engine & blockbasierte kollaborative Bearbeitung |
| **FLAW-02** | Storage | Dual-Storage Drift (SQLite vs. Dateisystem) | **Obsolet / Gelöst** | Kein Risiko: MindSift nutzt Pure-DB (keine `.md`-Dateien auf Platte) |
| **FLAW-03** | Netzwerk / WS | Proxy-Inkompatibilität bei WebSocket-Upgrades & Polling-Fallback | **Mittel** | Verzögerte Synchronisierung (1–3s Polling) in Firmennetzwerken |
| **FLAW-04** | Datenbank / Suche | Volltextsuche via `LIKE '%query%'` statt SQLite FTS5 | **Mittel** | Performance-Einbrüche bei wachsenden Notizbeständen (>10.000 Blöcke) |
@@ -23,13 +23,16 @@ Diese Datei dokumentiert eine ehrliche und schonungslose technische Analyse alle
## 🔍 Detaillierte Analyse & Maßnahmen
### FLAW-01: Kein echtes CRDT/OT – Lock-Heartbeat & Last-Write-Wins
- **Betroffene Komponenten:** [`api/websocket.md`](./api/websocket.md), [`modules/workspace.md`](./modules/workspace.md), `internal/ws/hub.go`, `frontend/src/features/workspace/WorkspaceView.tsx`
- **Ursache:** MindSift nutzt zur Vermeidung von Konflikten eine kooperative Bearbeitungssperre (*Document Lease Lock*, 30 Sekunden TTL mit Heartbeat). Wenn zwei Nutzer dasselbe Dokument geöffnet haben, blockiert der aktive Schreiber das Bearbeiten für andere. Verliert ein Nutzer jedoch kurzzeitig die WebSocket-Verbindung oder tippt während des Übergangs zum Polling-Fallback, werden Änderungen per REST via Last-Write-Wins (`PUT /api/v1/documents/:id`) gespeichert.
- **Folgen:** Wie in früheren Versionen beobachtet, können bei parallelem Schreiben Zeilen überschrieben werden oder Strukturhierarchien (Parent-IDs) verloren gehen.
- **Empfohlene Lösung:**
1. Langfristige Migration auf ein echtes blockweises CRDT-Framework (wie [Yjs](https://yjs.dev/) oder [Automerge](https://automerge.org/)).
2. Kurzfristig: Optimistisches Block-Locking (Sperre nur auf Block-Ebene `block_id` statt des gesamten Dokuments) und atomare Versionsprüfung (*Optimistic Concurrency Control* mit `version_seq`).
### FLAW-01: [GELÖST / IMPLEMENTIERT] Yjs Outliner CRDT & Multi-User Kollaboration
- **Status:** **Gelöst via Yjs CRDT Engine (Lösung i)**
- **Betroffene Komponenten:** [`api/websocket.md`](./api/websocket.md), [`modules/workspace.md`](./modules/workspace.md), `internal/ws/hub.go`, `internal/storage/sqlite.go`, `frontend/src/lib/crdt.ts`, `frontend/src/features/workspace/WorkspaceView.tsx`
- **Ursache (Historisch):** MindSift nutzte ein kooperatives Dokumenten-Lease-Locking mit Last-Write-Wins. Bei parallelem Zugriff oder Netztrennung konnten Änderungen überschrieben werden.
- **Implementierte Lösung (Yjs CRDT):**
1. **Block-Level Yjs CRDT:** Implementierung eines vollwertigen CRDT-Modells mit `Y.Doc`. Die Blockreihenfolge wird über ein `Y.Array<string>` (Block-IDs) konfliktfrei mit dem YATA-Algorithmus aufgelöst. Inhalte und Metadaten (Text, Level, Checked, Collapsed) sind in `Y.Map` gekapselt.
2. **Wegfall von Ganzdokument-Sperren:** Keine blockierenden Sperren mehr (`isLockedByOther` entfernt). Mehrere Nutzer können zeitgleich am selben Dokument schreiben.
3. **Dual-Persistence:** CRDT-Zustände und Delta-Updates werden binär/Base64 in SQLite (`document_crdt_state`, `document_crdt_updates`) persistiert. Zur Aufrechterhaltung relationaler Indizes (Global To-Dos, Tag-Indizierung, Suche) werden Blöcke parallel in `document_blocks` materialisiert.
4. **Live-Awareness & Presence:** Block-Fokus und Cursor-Zustand werden via `CRDT_AWARENESS` übertragen. Nutzer sehen ein Live-Badge im Header (`Live (N)`) sowie farbige Kollaborator-Pillen an aktiven Blöcken.
5. **REST- & Proxy-Sicherheit:** Ergänzend zu WebSockets stehen Endpunkte `GET /api/v1/documents/:id/crdt` und `PUT /api/v1/documents/:id/crdt` bereit.
---
+29 -22
View File
@@ -46,33 +46,40 @@ Der Bullet-Dot dient gleichzeitig als Drag-Anfasser. Wird ein Block verschoben,
---
## 💾 3. Pure-Database Architecture (Keine MD-Dateien auf Platte)
---
MindSift speichert Notizen und Outliner-Bäume **zu 100% relational in SQLite**. Es werden bewusst **keine Markdown-Dateien auf dem Dateisystem** abgelegt:
## ⚡ 3. Yjs CRDT Real-Time Engine & Dual-Persistence
Ab MindSift v2.1 setzt der Workspace auf ein **konfliktfreies Replicated Data Type (CRDT) Modell** auf Basis von [Yjs](https://yjs.dev/):
### Funktionsweise
1. **CRDT-Struktur im Client:**
- Ein `Y.Doc` pro geöffnetem Dokument.
- `Y.Array<string>` (`blocks`): Führt die Block-IDs in korrekter Reihenfolge. Der YATA-Algorithmus garantiert, dass simultanes Einfügen, Verschieben oder Löschen von Blöcken durch mehrere Nutzer deterministisch und konfliktfrei konvergiert.
- `Y.Map` (`block:<id>`): Speichert Inhalt (`content`), Einrückung (`level`), Erledigungsstatus (`checked`) und Faltungsstatus (`collapsed`).
2. **Kollaborative Bearbeitung ohne Sperren:**
- Nutzer können simultan dieselbe Notiz bearbeiten.
- Fokus- und Cursor-Positionen werden via `CRDT_AWARENESS` übertragen und als farbige Benutzer-Pillen direkt am jeweiligen Block visualisiert.
### Dual-Persistence in SQLite
```
Client (Block-Editor)
│ PUT /api/v1/documents/:id/blocks (oder WebSocket)
▼
Backend Engine
┌──────────────────────────────────────────────┐
│ Atomare Transaktion in SQLite: │
│ 1. DELETE FROM document_blocks WHERE doc_id │
│ 2. Batch INSERT der Blöcke: │
│ -> id, doc_id, content, level, checked, │
│ sort_order, collapsed │
│ 3. WebSocket Broadcast an Raum (doc:id) │
└──────────────────────────────────────────────┘
Client (Y.Doc Mutation)
│
├──▶ WebSocket (CRDT_UPDATE Base64 Delta)
│ │
│ ▼
│ SQLite: document_crdt_updates / document_crdt_state
│ (Mathematisch exakte CRDT-Konvergenz)
│
└──▶ Periodischer Snapshot / Flush
│
▼
SQLite: document_blocks
(Relationale Indizes für Global To-Dos, Tag-Explorer & FTS)
```
### Vorteile der reinen Datenbank-Architektur
1. **Kein Dual-Storage & kein Drift:**
- Keine Inkonsistenzen zwischen Festplattendateien und Datenbankzeilen.
- Keine Dateisystem-Sperren, Race-Conditions oder Berechtigungskonflikte auf Betriebssystemebene.
2. **Atomare Transaktionen (ACID):**
- Strukturänderungen (Einrücken, Unterbaum verschieben, Blöcke löschen) sind atomar in einer Transaktion gekapselt.
3. **Skalierbare Abfragen & Rollups:**
- Aggregationen (wie das globale To-Do Dashboard oder Tag-Suchen) erfolgen mit hochoptimierten SQL-Abfragen über relationale Indizes, anstatt Hunderte `.md`-Dateien von der Festplatte parsen zu müssen.
1. **Mathematische CRDT-Garantie:** Binäre Yjs-Updates werden in `document_crdt_updates` und periodisch als Snapshot in `document_crdt_state` gesichert. Auch nach einem Server-Neustart oder Offline-Phasen konvergieren alle Clients verlustfrei.
2. **Relationale Query-Fähigkeit:** Um schnelle SQL-Abfragen für das globale To-Do Rollup Dashboard und Hashtags zu ermöglichen, werden die Blöcke parallel in `document_blocks` materialisiert. Es existieren **keine Markdown-Dateien auf der Festplatte**.
---
+50 -1
View File
@@ -13,7 +13,8 @@
"lucide-react": "^0.468.0",
"prismjs": "^1.30.0",
"react": "^18.3.1",
"react-dom": "^18.3.1"
"react-dom": "^18.3.1",
"yjs": "^13.6.33"
},
"devDependencies": {
"@types/react": "^18.3.12",
@@ -1840,6 +1841,16 @@
"node": ">=0.12.0"
}
},
"node_modules/isomorphic.js": {
"version": "0.2.5",
"resolved": "https://registry.npmjs.org/isomorphic.js/-/isomorphic.js-0.2.5.tgz",
"integrity": "sha512-PIeMbHqMt4DnUP3MA/Flc0HElYjMXArsw1qwJZcm9sqR8mq3l8NYizFMty0pWwE/tzIGH3EKK5+jes5mAr85yw==",
"license": "MIT",
"funding": {
"type": "GitHub Sponsors ❤",
"url": "https://github.com/sponsors/dmonad"
}
},
"node_modules/jiti": {
"version": "1.21.7",
"resolved": "https://registry.npmjs.org/jiti/-/jiti-1.21.7.tgz",
@@ -1882,6 +1893,27 @@
"node": ">=6"
}
},
"node_modules/lib0": {
"version": "0.2.119",
"resolved": "https://registry.npmjs.org/lib0/-/lib0-0.2.119.tgz",
"integrity": "sha512-vh+TiejVy8Xm+hCdnRfn9BthGNvNUZdEXltNjVZL3TzRIKP3s1lnQh8pWd/Q0wCgY0FPAGVVjFF4ZiHUV3LFgA==",
"license": "MIT",
"dependencies": {
"isomorphic.js": "^0.2.4"
},
"bin": {
"0ecdsa-generate-keypair": "bin/0ecdsa-generate-keypair.js",
"0gentesthtml": "bin/gentesthtml.js",
"0serve": "bin/0serve.js"
},
"engines": {
"node": ">=16"
},
"funding": {
"type": "GitHub Sponsors ❤",
"url": "https://github.com/sponsors/dmonad"
}
},
"node_modules/lilconfig": {
"version": "3.1.4",
"resolved": "https://registry.npmjs.org/lilconfig/-/lilconfig-3.1.4.tgz",
@@ -2756,6 +2788,23 @@
"integrity": "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g==",
"dev": true,
"license": "ISC"
},
"node_modules/yjs": {
"version": "13.6.33",
"resolved": "https://registry.npmjs.org/yjs/-/yjs-13.6.33.tgz",
"integrity": "sha512-q/UYvr1gOk4XII+Cu1CW2dINjYqJJFifTzogEaIARznrvX7uOlHsdTEXcnOBMuIsrCGEEKdNxi74kyMLZlKVfw==",
"license": "MIT",
"dependencies": {
"lib0": "^0.2.99"
},
"engines": {
"node": ">=16.0.0",
"npm": ">=8.0.0"
},
"funding": {
"type": "GitHub Sponsors ❤",
"url": "https://github.com/sponsors/dmonad"
}
}
}
}
+2 -1
View File
@@ -16,7 +16,8 @@
"lucide-react": "^0.468.0",
"prismjs": "^1.30.0",
"react": "^18.3.1",
"react-dom": "^18.3.1"
"react-dom": "^18.3.1",
"yjs": "^13.6.33"
},
"devDependencies": {
"@types/react": "^18.3.12",
+175 -19
View File
@@ -55,6 +55,7 @@ import { CodeBlockItem } from './CodeBlockItem';
import { UserAutocomplete } from '../UserAutocomplete';
import { UnifiedShareModal } from '../../components/UnifiedShareModal';
import { useMobileActions } from '../../context/MobileActionContext';
import { DocumentCRDTSession, CollaboratorAwareness, getCollaboratorColor } from '../../lib/crdt';
interface WorkspaceViewProps {
currentUser?: User | null;
@@ -216,7 +217,11 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
const [showFolderShareModal, setShowFolderShareModal] = useState(false);
const [folderShares, setFolderShares] = useState<FolderShare[]>([]);
// --- DOCUMENT LOCKING STATE ---
// --- DOCUMENT CRDT & COLLABORATION STATE ---
const crdtSessionRef = useRef<DocumentCRDTSession | null>(null);
const [activePeers, setActivePeers] = useState<CollaboratorAwareness[]>([]);
// --- DOCUMENT LOCKING STATE (FALLBACK) ---
const [activeLock, setActiveLock] = useState<DocLockInfo | null>(null);
const [holdsLock, setHoldsLock] = useState(false);
const holdsLockRef = useRef(false);
@@ -228,8 +233,9 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
const blockInputRefs = useRef<Record<string, HTMLTextAreaElement | HTMLInputElement | null>>({});
const isLockedByOther = !!(activeLock && activeLock.userId !== currentUser?.id);
const isReadOnly = selectedDoc?.permission === 'read' || isLockedByOther;
// With CRDT, multiple users can write simultaneously without lockouts
const isLockedByOther = false;
const isReadOnly = selectedDoc?.permission === 'read';
// Track server-synced blocks & metadata per document, plus local unsaved block deltas
const syncedDocMetaRef = useRef<Map<string, { title: string; icon: string; tagsJson: string; isPinned: boolean }>>(new Map());
@@ -418,12 +424,34 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
return false;
};
const handleBlockFocus = () => {
const broadcastAwareness = (blockId: string | null) => {
if (!selectedDoc || !wsRef.current || wsRef.current.readyState !== WebSocket.OPEN) return;
try {
wsRef.current.send(
JSON.stringify({
event: 'CRDT_AWARENESS',
docId: selectedDoc.id,
payload: {
userId: currentUser?.id,
userName: currentUser?.displayName || currentUser?.username || 'Kollege',
activeBlockId: blockId,
color: getCollaboratorColor(currentUser?.id || ''),
lastActive: Date.now(),
},
})
);
} catch {}
};
const handleBlockFocus = (blockId?: string) => {
isEditorFocusedRef.current = true;
if (idleUnlockTimerRef.current) {
clearTimeout(idleUnlockTimerRef.current);
idleUnlockTimerRef.current = null;
}
if (blockId) {
broadcastAwareness(blockId);
}
if (!isReadOnly) {
ensureLock();
}
@@ -431,6 +459,7 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
const handleBlockBlur = () => {
isEditorFocusedRef.current = false;
broadcastAwareness(null);
if (idleUnlockTimerRef.current) {
clearTimeout(idleUnlockTimerRef.current);
}
@@ -611,6 +640,11 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
}
});
// Sync into active CRDT session
if (crdtSessionRef.current && crdtSessionRef.current.docId === updatedDoc.id) {
crdtSessionRef.current.syncFullBlocks(updatedDoc.blocks || []);
}
// 1. Update local UI state immediately for responsive, instant typing
setSelectedDoc(updatedDoc);
setDocs((prev) => prev.map((d) => (d.id === updatedDoc.id ? updatedDoc : d)));
@@ -732,16 +766,73 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
const currentUserIdRef = useRef<string | null>(null);
currentUserIdRef.current = currentUser?.id || null;
// Handle document switch and lock status checks
// Handle document switch, CRDT session lifecycle, and lock status checks
useEffect(() => {
const prevId = currentDocIdRef.current;
if (prevId && prevId !== selectedDoc?.id) {
flushPendingSave();
releaseMyLock(prevId);
if (wsRef.current && wsRef.current.readyState === WebSocket.OPEN) {
try {
wsRef.current.send(JSON.stringify({ event: 'CRDT_LEAVE', docId: prevId }));
} catch {}
}
if (crdtSessionRef.current) {
crdtSessionRef.current.destroy();
crdtSessionRef.current = null;
}
setActivePeers([]);
}
currentDocIdRef.current = selectedDoc?.id || null;
if (selectedDoc) {
const targetDoc = selectedDoc;
const session = new DocumentCRDTSession(targetDoc.id);
session.initFromBlocks(targetDoc.blocks || []);
session.onLocalUpdate = (b64Update) => {
if (wsRef.current && wsRef.current.readyState === WebSocket.OPEN) {
try {
wsRef.current.send(
JSON.stringify({
event: 'CRDT_UPDATE',
docId: targetDoc.id,
payload: { update: b64Update },
})
);
} catch {}
}
};
session.onRemoteChange = (crdtBlocks) => {
setSelectedDoc((prev) => (prev && prev.id === targetDoc.id ? { ...prev, blocks: crdtBlocks } : prev));
setDocs((prev) => prev.map((d) => (d.id === targetDoc.id ? { ...d, blocks: crdtBlocks } : d)));
};
crdtSessionRef.current = session;
if (wsRef.current && wsRef.current.readyState === WebSocket.OPEN) {
try {
wsRef.current.send(JSON.stringify({ event: 'CRDT_JOIN', docId: targetDoc.id }));
} catch {}
}
api
.getDocumentCRDT(targetDoc.id)
.then((res) => {
if (res && crdtSessionRef.current && crdtSessionRef.current.docId === targetDoc.id) {
if (res.state) {
crdtSessionRef.current.applyRemoteUpdate(res.state);
}
if (res.updates && Array.isArray(res.updates)) {
crdtSessionRef.current.applyRemoteUpdates(res.updates);
}
const blocks = crdtSessionRef.current.getBlocks();
if (blocks.length > 0) {
setSelectedDoc((prev) => (prev && prev.id === targetDoc.id ? { ...prev, blocks } : prev));
setDocs((prev) => prev.map((d) => (d.id === targetDoc.id ? { ...d, blocks } : d)));
}
}
})
.catch(() => {});
api
.getDocLock(selectedDoc.id)
.then((res) => {
@@ -759,6 +850,15 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
return () => {
flushPendingSave();
if (currentDocIdRef.current) {
if (wsRef.current && wsRef.current.readyState === WebSocket.OPEN) {
try {
wsRef.current.send(JSON.stringify({ event: 'CRDT_LEAVE', docId: currentDocIdRef.current }));
} catch {}
}
if (crdtSessionRef.current) {
crdtSessionRef.current.destroy();
crdtSessionRef.current = null;
}
releaseMyLock(currentDocIdRef.current);
}
};
@@ -819,7 +919,47 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
const handleWsMsg = (msg: any) => {
if (!msg || !msg.event) return;
if (msg.event === 'DOC_LOCKED') {
if (msg.event === 'CRDT_INIT') {
const payload = msg.payload as { docId: string; state?: string; updates?: string[] };
if (payload && payload.docId === selectedDocIdRef.current && crdtSessionRef.current) {
if (payload.state) {
crdtSessionRef.current.applyRemoteUpdate(payload.state);
}
if (payload.updates && Array.isArray(payload.updates)) {
crdtSessionRef.current.applyRemoteUpdates(payload.updates);
}
const blocks = crdtSessionRef.current.getBlocks();
if (blocks.length > 0) {
setSelectedDoc((prev) => (prev && prev.id === payload.docId ? { ...prev, blocks } : prev));
setDocs((prev) => prev.map((d) => (d.id === payload.docId ? { ...d, blocks } : d)));
}
}
} else if (msg.event === 'CRDT_UPDATE') {
const payload = msg.payload as { docId: string; update: string; fromUserId?: string };
if (payload && payload.docId === selectedDocIdRef.current && crdtSessionRef.current) {
if (payload.fromUserId !== currentUserIdRef.current) {
crdtSessionRef.current.applyRemoteUpdate(payload.update);
const blocks = crdtSessionRef.current.getBlocks();
if (blocks && blocks.length > 0) {
setSelectedDoc((prev) => (prev && prev.id === payload.docId ? { ...prev, blocks } : prev));
setDocs((prev) => prev.map((d) => (d.id === payload.docId ? { ...d, blocks } : d)));
}
}
}
} else if (msg.event === 'CRDT_AWARENESS') {
const payload = (msg.payload?.payload || msg.payload) as CollaboratorAwareness;
if (payload && payload.userId && payload.userId !== currentUserIdRef.current) {
setActivePeers((prev) => {
const filtered = prev.filter((c) => c.userId !== payload.userId);
return [...filtered, { ...payload, color: getCollaboratorColor(payload.userId), lastActive: Date.now() }];
});
}
} else if (msg.event === 'CRDT_AWARENESS_LEAVE') {
const payload = msg.payload as { docId?: string; userId?: string };
if (payload && payload.userId) {
setActivePeers((prev) => prev.filter((c) => c.userId !== payload.userId));
}
} else if (msg.event === 'DOC_LOCKED') {
const payload = msg.payload as DocLockInfo;
if (payload && payload.docId === selectedDocIdRef.current) {
if (payload.userId !== currentUserIdRef.current) {
@@ -975,6 +1115,11 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
wsReconnectDelayRef.current = 2000;
setWsDisconnectReason(null);
setWsConnected(true);
if (selectedDocIdRef.current) {
try {
ws.send(JSON.stringify({ event: 'CRDT_JOIN', docId: selectedDocIdRef.current }));
} catch {}
}
api.getFolders().then(setCustomFolders).catch(() => {});
api.getSharedFolders().then(setSharedFolders).catch(() => {});
};
@@ -4215,22 +4360,21 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
</button>
)}
{holdsLock && (
{activePeers.length > 0 ? (
<span
className="px-2 py-0.5 rounded text-[10px] font-semibold border border-emerald-500/30 bg-emerald-500/15 text-emerald-600 dark:text-emerald-400 flex items-center gap-1.5 animate-pulse"
title={`Live-Kollaboration via CRDT aktiv: ${activePeers.map((c) => c.userName).join(', ')}`}
>
<span className="w-1.5 h-1.5 rounded-full bg-emerald-500 animate-ping" />
<Users className="w-2.5 h-2.5" />
<span>Live ({activePeers.length})</span>
</span>
) : holdsLock ? (
<span className="px-2 py-0.5 rounded text-[10px] font-semibold border border-emerald-500/30 bg-emerald-500/15 text-emerald-600 dark:text-emerald-400 flex items-center gap-1">
<span className="w-1.5 h-1.5 rounded-full bg-emerald-500 animate-pulse"></span>
<span>Schreibzugriff aktiv</span>
<span>CRDT Sync aktiv</span>
</span>
)}
{isLockedByOther && (
<span
className="px-2 py-0.5 rounded text-[10px] font-semibold border border-amber-500/30 bg-amber-500/15 text-amber-600 dark:text-amber-400 flex items-center gap-1 animate-pulse cursor-help"
title={`Diese Notiz wird aktuell von ${activeLock?.userName || 'einem anderen Benutzer'} bearbeitet. Schreibzugriff vorübergehend gesperrt.`}
>
<Lock className="w-2.5 h-2.5" />
<span>Wird bearbeitet ({activeLock?.userName || 'anderer Nutzer'})</span>
</span>
)}
) : null}
{selectedDoc.sharedWith && selectedDoc.sharedWith.length > 0 && selectedDoc.isOwner !== false && (
<button
@@ -4590,6 +4734,18 @@ export const WorkspaceView: React.FC<WorkspaceViewProps> = ({
{/* Block Content Container */}
<div className="flex-1 min-w-0 pt-0.5 relative">
{/* Live Collaborator Presence Indicator */}
{activePeers.filter((c) => c.activeBlockId === block.id).map((c) => (
<div
key={c.userId}
className="absolute -top-3.5 right-1 text-[10px] px-1.5 py-0.5 rounded font-semibold text-white shadow-xs pointer-events-none z-10 flex items-center gap-1 animate-fadeIn"
style={{ backgroundColor: c.color }}
>
<span className="w-1.5 h-1.5 rounded-full bg-white animate-ping" />
<span>{c.userName}</span>
</div>
))}
{/* TYPE: TODO */}
{block.type === 'todo' && (
<div className="space-y-1">
+11
View File
@@ -602,6 +602,17 @@ export const api = {
};
},
async getDocumentCRDT(id: string): Promise<{ docId: string; state?: string; updates?: string[] }> {
return request<{ docId: string; state?: string; updates?: string[] }>(`/api/v1/documents/${id}/crdt`);
},
async saveDocumentCRDT(id: string, state: string): Promise<{ success: boolean }> {
return request<{ success: boolean }>(`/api/v1/documents/${id}/crdt`, {
method: 'PUT',
body: JSON.stringify({ state }),
});
},
async deleteDocument(id: string): Promise<void> {
await request(`/api/v1/documents/${id}`, { method: 'DELETE' });
},
+271
View File
@@ -0,0 +1,271 @@
import * as Y from 'yjs';
import { BlockType, DocumentBlock } from './types';
// Convert Uint8Array to Base64
export function uint8ArrayToBase64(bytes: Uint8Array): string {
let binary = '';
const len = bytes.byteLength;
for (let i = 0; i < len; i++) {
binary += String.fromCharCode(bytes[i]);
}
return btoa(binary);
}
// Convert Base64 to Uint8Array
export function base64ToUint8Array(base64: string): Uint8Array {
const binaryString = atob(base64);
const len = binaryString.length;
const bytes = new Uint8Array(len);
for (let i = 0; i < len; i++) {
bytes[i] = binaryString.charCodeAt(i);
}
return bytes;
}
const COLLAB_COLORS = [
'#3B82F6', // Blue
'#10B981', // Emerald
'#F59E0B', // Amber
'#8B5CF6', // Purple
'#EC4899', // Pink
'#06B6D4', // Cyan
'#F97316', // Orange
'#14B8A6', // Teal
];
export function getCollaboratorColor(userId: string): string {
if (!userId) return COLLAB_COLORS[0];
let hash = 0;
for (let i = 0; i < userId.length; i++) {
hash = (hash << 5) - hash + userId.charCodeAt(i);
hash |= 0;
}
const idx = Math.abs(hash) % COLLAB_COLORS.length;
return COLLAB_COLORS[idx];
}
export interface CollaboratorAwareness {
userId: string;
userName: string;
activeBlockId: string | null;
color: string;
lastActive: number;
}
export class DocumentCRDTSession {
public docId: string;
public doc: Y.Doc;
private order: Y.Array<string>;
private blocks: Y.Map<Y.Map<any>>;
private destroyed: boolean = false;
public onLocalUpdate?: (base64Update: string) => void;
public onRemoteChange?: (blocks: DocumentBlock[]) => void;
constructor(docId: string) {
this.docId = docId;
this.doc = new Y.Doc({ guid: docId });
this.order = this.doc.getArray<string>('blockOrder');
this.blocks = this.doc.getMap<Y.Map<any>>('blocks');
this.doc.on('update', (update: Uint8Array, origin: any) => {
if (this.destroyed) return;
if (origin !== 'remote') {
const b64 = uint8ArrayToBase64(update);
if (this.onLocalUpdate) {
this.onLocalUpdate(b64);
}
} else {
if (this.onRemoteChange) {
this.onRemoteChange(this.getBlocks());
}
}
});
}
public destroy(): void {
this.destroyed = true;
this.doc.destroy();
}
public getBlocks(): DocumentBlock[] {
const orderList = this.order.toArray();
const result: DocumentBlock[] = [];
for (const id of orderList) {
const bMap = this.blocks.get(id);
if (bMap) {
result.push({
id,
docId: this.docId,
type: (bMap.get('type') || 'bullet') as BlockType,
content: bMap.get('content') || '',
level: typeof bMap.get('level') === 'number' ? bMap.get('level') : 0,
checked: !!bMap.get('checked'),
collapsed: !!bMap.get('collapsed'),
lang: bMap.get('lang') || '',
sortOrder: typeof bMap.get('sortOrder') === 'number' ? bMap.get('sortOrder') : 0,
updatedAt: bMap.get('updatedAt') || undefined,
});
}
}
return result;
}
public initFromBlocks(initialBlocks: DocumentBlock[]): void {
this.doc.transact(() => {
if (this.order.length === 0 && initialBlocks.length > 0) {
const ids: string[] = [];
for (const b of initialBlocks) {
ids.push(b.id);
const bMap = new Y.Map<any>();
bMap.set('id', b.id);
bMap.set('type', b.type || 'bullet');
bMap.set('content', b.content || '');
bMap.set('level', b.level || 0);
bMap.set('checked', !!b.checked);
bMap.set('collapsed', !!b.collapsed);
bMap.set('lang', b.lang || '');
bMap.set('sortOrder', b.sortOrder || 0);
bMap.set('updatedAt', b.updatedAt || new Date().toISOString());
this.blocks.set(b.id, bMap);
}
this.order.push(ids);
}
}, 'init');
}
public updateBlock(id: string, updates: Partial<DocumentBlock>): void {
this.doc.transact(() => {
const bMap = this.blocks.get(id);
if (bMap) {
if (updates.type !== undefined) bMap.set('type', updates.type);
if (updates.content !== undefined) bMap.set('content', updates.content);
if (updates.level !== undefined) bMap.set('level', updates.level);
if (updates.checked !== undefined) bMap.set('checked', updates.checked);
if (updates.collapsed !== undefined) bMap.set('collapsed', updates.collapsed);
if (updates.lang !== undefined) bMap.set('lang', updates.lang);
if (updates.sortOrder !== undefined) bMap.set('sortOrder', updates.sortOrder);
bMap.set('updatedAt', new Date().toISOString());
}
}, 'local');
}
public insertBlock(index: number, block: DocumentBlock): void {
this.doc.transact(() => {
const bMap = new Y.Map<any>();
bMap.set('id', block.id);
bMap.set('type', block.type || 'bullet');
bMap.set('content', block.content || '');
bMap.set('level', block.level || 0);
bMap.set('checked', !!block.checked);
bMap.set('collapsed', !!block.collapsed);
bMap.set('lang', block.lang || '');
bMap.set('sortOrder', block.sortOrder || 0);
bMap.set('updatedAt', new Date().toISOString());
this.blocks.set(block.id, bMap);
if (index >= 0 && index <= this.order.length) {
this.order.insert(index, [block.id]);
} else {
this.order.push([block.id]);
}
}, 'local');
}
public deleteBlock(id: string): void {
this.doc.transact(() => {
const currentOrder = this.order.toArray();
const idx = currentOrder.indexOf(id);
if (idx !== -1) {
this.order.delete(idx, 1);
}
this.blocks.delete(id);
}, 'local');
}
public deleteMultipleBlocks(ids: string[]): void {
if (ids.length === 0) return;
const toDeleteSet = new Set(ids);
this.doc.transact(() => {
const currentOrder = this.order.toArray();
// Delete from order backwards to preserve indices
for (let i = currentOrder.length - 1; i >= 0; i--) {
if (toDeleteSet.has(currentOrder[i])) {
this.order.delete(i, 1);
}
}
for (const id of ids) {
this.blocks.delete(id);
}
}, 'local');
}
public reorderBlocks(newOrder: string[]): void {
this.doc.transact(() => {
this.order.delete(0, this.order.length);
this.order.push(newOrder);
}, 'local');
}
public syncFullBlocks(blocks: DocumentBlock[]): void {
this.doc.transact(() => {
const nextIds = blocks.map((b) => b.id);
const currentIds = new Set(this.order.toArray());
// Upsert all blocks
for (const b of blocks) {
let bMap = this.blocks.get(b.id);
if (!bMap) {
bMap = new Y.Map<any>();
this.blocks.set(b.id, bMap);
}
bMap.set('id', b.id);
bMap.set('type', b.type || 'bullet');
bMap.set('content', b.content || '');
bMap.set('level', b.level || 0);
bMap.set('checked', !!b.checked);
bMap.set('collapsed', !!b.collapsed);
bMap.set('lang', b.lang || '');
bMap.set('sortOrder', b.sortOrder || 0);
bMap.set('updatedAt', b.updatedAt || new Date().toISOString());
}
// Delete blocks no longer present
const nextIdSet = new Set(nextIds);
currentIds.forEach((id) => {
if (!nextIdSet.has(id)) {
this.blocks.delete(id);
}
});
// Update order if different
const currentOrder = this.order.toArray();
const orderChanged =
currentOrder.length !== nextIds.length || currentOrder.some((id, idx) => id !== nextIds[idx]);
if (orderChanged) {
this.order.delete(0, this.order.length);
this.order.push(nextIds);
}
}, 'local');
}
public applyRemoteUpdate(base64Update: string): void {
try {
const bytes = base64ToUint8Array(base64Update);
Y.applyUpdate(this.doc, bytes, 'remote');
} catch (err) {
console.error('[CRDT] Failed to apply remote update:', err);
}
}
public applyRemoteUpdates(base64Updates: string[]): void {
for (const b64 of base64Updates) {
this.applyRemoteUpdate(b64);
}
}
public encodeState(): string {
const bytes = Y.encodeStateAsUpdate(this.doc);
return uint8ArrayToBase64(bytes);
}
}
+70
View File
@@ -3,6 +3,7 @@ package api
import (
"bytes"
"crypto/tls"
"encoding/base64"
"encoding/json"
"fmt"
"io"
@@ -30,6 +31,9 @@ type Server struct {
}
func NewServer(cfg *config.Config, store *storage.Store, wsHub *ws.Hub, staticFS fs.FS) *Server {
if wsHub != nil && store != nil {
wsHub.SetCRDTStore(store)
}
srv := &Server{
cfg: cfg,
store: store,
@@ -141,6 +145,10 @@ func (s *Server) Router() http.Handler {
r.Post("/api/v1/documents/{id}/lock/heartbeat", s.handleHeartbeatDocLock)
r.Delete("/api/v1/documents/{id}/lock", s.handleUnlockDoc)
// Document CRDT Realtime Sync
r.Get("/api/v1/documents/{id}/crdt", s.handleGetDocCRDT)
r.Put("/api/v1/documents/{id}/crdt", s.handleSaveDocCRDT)
// Shared Lists & Realtime Collaboration
r.Get("/api/v1/lists", s.handleGetAllLists)
r.Post("/api/v1/lists", s.handleCreateList)
@@ -1373,6 +1381,68 @@ func (s *Server) handleUnlockDoc(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, map[string]bool{"unlocked": released})
}
// --- DOCUMENT CRDT REALTIME SYNC ---
func (s *Server) handleGetDocCRDT(w http.ResponseWriter, r *http.Request) {
userID := auth.GetUserID(r.Context())
docID := chi.URLParam(r, "id")
if _, err := s.store.GetDocumentByID(userID, docID); err != nil {
writeError(w, http.StatusNotFound, "Dokument nicht gefunden")
return
}
stateBlob, updates, err := s.store.GetDocumentCRDTState(docID)
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
b64State := ""
if len(stateBlob) > 0 {
b64State = base64.StdEncoding.EncodeToString(stateBlob)
}
var b64Updates []string
for _, u := range updates {
b64Updates = append(b64Updates, base64.StdEncoding.EncodeToString(u))
}
writeJSON(w, http.StatusOK, map[string]interface{}{
"docId": docID,
"state": b64State,
"updates": b64Updates,
})
}
func (s *Server) handleSaveDocCRDT(w http.ResponseWriter, r *http.Request) {
userID := auth.GetUserID(r.Context())
docID := chi.URLParam(r, "id")
doc, err := s.store.GetDocumentByID(userID, docID)
if err != nil {
writeError(w, http.StatusNotFound, "Dokument nicht gefunden")
return
}
if !doc.IsOwner && doc.Permission != "write" {
writeError(w, http.StatusForbidden, "Keine Schreibberechtigung")
return
}
var req struct {
State string `json:"state"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "Ungültiger Request-Body")
return
}
if req.State != "" {
raw, err := base64.StdEncoding.DecodeString(req.State)
if err != nil {
writeError(w, http.StatusBadRequest, "Ungültiges Base64 Format")
return
}
if err := s.store.SaveDocumentCRDTState(docID, raw, userID); err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
}
writeJSON(w, http.StatusOK, map[string]bool{"success": true})
}
// --- REALTIME LISTS & SHARING ---
func (s *Server) handleGetAllLists(w http.ResponseWriter, r *http.Request) {
+70
View File
@@ -838,3 +838,73 @@ func TestWebhookTriageBadgeAndIngestion(t *testing.T) {
t.Fatalf("expected thought content to contain 'Waschmaschine', got '%s'", ingestResp.Thought.Content)
}
}
func TestDocumentCRDTEndpoints(t *testing.T) {
tmpDir := t.TempDir()
dbPath := filepath.Join(tmpDir, "crdt_api_test.db")
store, err := storage.NewStore(dbPath)
if err != nil {
t.Fatalf("NewStore failed: %v", err)
}
defer store.Close()
cfg := &config.Config{
JWTSecret: "test-jwt-secret-key",
AllowRegistration: true,
}
hub := ws.NewHub()
srv := NewServer(cfg, store, hub, nil)
router := srv.Router()
user, err := store.CreateUser("crdt_api@example.com", "hash", "CRDT API User", "member")
if err != nil {
t.Fatalf("CreateUser failed: %v", err)
}
token, _ := auth.GenerateToken(cfg.JWTSecret, user.ID, user.Email, user.Role)
doc, err := store.CreateDocument(user.ID, "", "API CRDT Doc", "📄", "", nil, nil, false)
if err != nil {
t.Fatalf("CreateDocument failed: %v", err)
}
// 1. Initial GET /api/v1/documents/{id}/crdt
req := httptest.NewRequest(http.MethodGet, "/api/v1/documents/"+doc.ID+"/crdt", nil)
req.Header.Set("Authorization", "Bearer "+token)
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected 200 OK from GET crdt, got %d: %s", rec.Code, rec.Body.String())
}
// 2. PUT /api/v1/documents/{id}/crdt with base64 state
savePayload := []byte(`{"state":"AQIDBA=="}`)
req = httptest.NewRequest(http.MethodPut, "/api/v1/documents/"+doc.ID+"/crdt", bytes.NewReader(savePayload))
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "application/json")
rec = httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected 200 OK from PUT crdt, got %d: %s", rec.Code, rec.Body.String())
}
// 3. Verify state persisted
req = httptest.NewRequest(http.MethodGet, "/api/v1/documents/"+doc.ID+"/crdt", nil)
req.Header.Set("Authorization", "Bearer "+token)
rec = httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected 200 OK from GET crdt, got %d", rec.Code)
}
var crdtResp struct {
DocID string `json:"docId"`
State string `json:"state"`
}
_ = json.Unmarshal(rec.Body.Bytes(), &crdtResp)
if crdtResp.State != "AQIDBA==" {
t.Fatalf("expected state 'AQIDBA==', got '%s'", crdtResp.State)
}
}
+72
View File
@@ -232,6 +232,25 @@ func (s *Store) migrate() error {
PRIMARY KEY(user_id, tag)
);
CREATE TABLE IF NOT EXISTS document_crdt_state (
doc_id TEXT PRIMARY KEY,
state_blob BLOB NOT NULL,
updated_at TEXT NOT NULL,
updated_by TEXT NOT NULL DEFAULT '',
FOREIGN KEY(doc_id) REFERENCES documents(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS document_crdt_updates (
id INTEGER PRIMARY KEY AUTOINCREMENT,
doc_id TEXT NOT NULL,
update_blob BLOB NOT NULL,
created_at TEXT NOT NULL,
created_by TEXT NOT NULL DEFAULT '',
FOREIGN KEY(doc_id) REFERENCES documents(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_doc_crdt_updates_doc ON document_crdt_updates(doc_id);
CREATE TABLE IF NOT EXISTS lists (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
@@ -1559,6 +1578,59 @@ func (s *Store) PatchDocumentBlocks(userID, docID string, req domain.DocBlocksPa
return doc, mergedOrder, nil
}
// GetDocumentCRDTState returns the compacted CRDT state blob and any pending updates for a document.
func (s *Store) GetDocumentCRDTState(docID string) ([]byte, [][]byte, error) {
var stateBlob []byte
err := s.db.QueryRow(`SELECT state_blob FROM document_crdt_state WHERE doc_id = ?`, docID).Scan(&stateBlob)
if err != nil && err != sql.ErrNoRows {
return nil, nil, err
}
rows, err := s.db.Query(`SELECT update_blob FROM document_crdt_updates WHERE doc_id = ? ORDER BY id ASC`, docID)
if err != nil {
return stateBlob, nil, nil
}
defer rows.Close()
var updates [][]byte
for rows.Next() {
var u []byte
if err := rows.Scan(&u); err == nil && len(u) > 0 {
updates = append(updates, u)
}
}
return stateBlob, updates, nil
}
// AppendDocumentCRDTUpdate appends a new CRDT update delta for a document.
func (s *Store) AppendDocumentCRDTUpdate(docID string, update []byte, userID string) error {
now := domain.NowISO()
_, err := s.db.Exec(`
INSERT INTO document_crdt_updates (doc_id, update_blob, created_at, created_by)
VALUES (?, ?, ?, ?)
`, docID, update, now, userID)
return err
}
// SaveDocumentCRDTState saves a compacted snapshot of the document CRDT state and prunes older incremental updates.
func (s *Store) SaveDocumentCRDTState(docID string, stateBlob []byte, userID string) error {
now := domain.NowISO()
_, err := s.db.Exec(`
INSERT INTO document_crdt_state (doc_id, state_blob, updated_at, updated_by)
VALUES (?, ?, ?, ?)
ON CONFLICT(doc_id) DO UPDATE SET
state_blob = excluded.state_blob,
updated_at = excluded.updated_at,
updated_by = excluded.updated_by
`, docID, stateBlob, now, userID)
if err != nil {
return err
}
// Prune incremental updates older than the snapshot
_, _ = s.db.Exec(`DELETE FROM document_crdt_updates WHERE doc_id = ?`, docID)
return nil
}
// GetAuthorizedDocUsers returns all user IDs who have read/write/owner access to docID.
func (s *Store) GetAuthorizedDocUsers(docID string) ([]string, error) {
userSet := make(map[string]bool)
+59
View File
@@ -683,3 +683,62 @@ func TestDatabaseFirstWorkspaceArchitecture(t *testing.T) {
}
}
func TestDocumentCRDTState(t *testing.T) {
tmpDir := t.TempDir()
dbPath := filepath.Join(tmpDir, "crdt_test.db")
store, err := NewStore(dbPath)
if err != nil {
t.Fatalf("NewStore failed: %v", err)
}
defer store.Close()
user, err := store.CreateUser("crdt@example.com", "hash", "CRDT User", "member")
if err != nil {
t.Fatalf("CreateUser failed: %v", err)
}
doc, err := store.CreateDocument(user.ID, "", "CRDT Test Doc", "📝", "", nil, nil, false)
if err != nil {
t.Fatalf("CreateDocument failed: %v", err)
}
// 1. Initial CRDT state is empty
state, updates, err := store.GetDocumentCRDTState(doc.ID)
if err != nil {
t.Fatalf("GetDocumentCRDTState failed: %v", err)
}
if len(state) != 0 || len(updates) != 0 {
t.Fatalf("expected empty CRDT state, got state=%d updates=%d", len(state), len(updates))
}
// 2. Append incremental updates
u1 := []byte{1, 2, 3, 4}
u2 := []byte{5, 6, 7, 8}
if err := store.AppendDocumentCRDTUpdate(doc.ID, u1, user.ID); err != nil {
t.Fatalf("AppendDocumentCRDTUpdate 1 failed: %v", err)
}
if err := store.AppendDocumentCRDTUpdate(doc.ID, u2, user.ID); err != nil {
t.Fatalf("AppendDocumentCRDTUpdate 2 failed: %v", err)
}
_, updatesAfter, err := store.GetDocumentCRDTState(doc.ID)
if err != nil || len(updatesAfter) != 2 {
t.Fatalf("expected 2 updates, got %d (err: %v)", len(updatesAfter), err)
}
// 3. Compact snapshot state
snapshot := []byte{99, 100, 101}
if err := store.SaveDocumentCRDTState(doc.ID, snapshot, user.ID); err != nil {
t.Fatalf("SaveDocumentCRDTState failed: %v", err)
}
stateAfter, updatesAfterCompact, err := store.GetDocumentCRDTState(doc.ID)
if err != nil {
t.Fatalf("GetDocumentCRDTState after compact failed: %v", err)
}
if len(stateAfter) != 3 || len(updatesAfterCompact) != 0 {
t.Fatalf("expected compacted state of length 3 and 0 updates, got state=%d updates=%d", len(stateAfter), len(updatesAfterCompact))
}
}
+173 -9
View File
@@ -1,9 +1,11 @@
package ws
import (
"encoding/base64"
"encoding/json"
"log"
"net/http"
"strings"
"sync"
"time"
@@ -12,19 +14,20 @@ import (
)
var upgrader = websocket.Upgrader{
ReadBufferSize: 2048,
WriteBufferSize: 2048,
ReadBufferSize: 4096,
WriteBufferSize: 4096,
CheckOrigin: func(r *http.Request) bool {
return true // Self-hosted reverse proxy compatible
},
}
type Client struct {
hub *Hub
conn *websocket.Conn
roomID string
userID string
send chan []byte
hub *Hub
conn *websocket.Conn
roomID string
userID string
send chan []byte
subscribedRooms map[string]bool
}
type docLockItem struct {
@@ -35,12 +38,19 @@ type docLockItem struct {
expiresAt time.Time
}
type CRDTStore interface {
GetDocumentCRDTState(docID string) ([]byte, [][]byte, error)
AppendDocumentCRDTUpdate(docID string, update []byte, userID string) error
SaveDocumentCRDTState(docID string, stateBlob []byte, userID string) error
}
type Hub struct {
mu sync.RWMutex
rooms map[string]map[*Client]bool
userRooms map[string]map[*Client]bool
allClients map[*Client]bool
locks map[string]*docLockItem
crdtStore CRDTStore
}
func NewHub() *Hub {
@@ -54,6 +64,12 @@ func NewHub() *Hub {
return h
}
func (h *Hub) SetCRDTStore(store CRDTStore) {
h.mu.Lock()
defer h.mu.Unlock()
h.crdtStore = store
}
func (h *Hub) Register(c *Client) {
h.mu.Lock()
h.allClients[c] = true
@@ -90,6 +106,22 @@ func (h *Hub) Unregister(c *Client) {
}
}
}
var docRoomsLeft []string
if c.subscribedRooms != nil {
for rID := range c.subscribedRooms {
if room, ok := h.rooms[rID]; ok {
delete(room, c)
if len(room) == 0 {
delete(h.rooms, rID)
}
}
if strings.HasPrefix(rID, "doc:") {
docRoomsLeft = append(docRoomsLeft, strings.TrimPrefix(rID, "doc:"))
}
}
}
userRemaining := 0
if c.userID != "" && c.userID != "anonymous" {
if uRoom, ok := h.userRooms[c.userID]; ok {
@@ -119,6 +151,14 @@ func (h *Hub) Unregister(c *Client) {
for _, docID := range releasedDocIDs {
h.BroadcastToAll("DOC_UNLOCKED", map[string]string{"docId": docID, "doc_id": docID})
}
// Broadcast awareness leave for doc rooms
for _, docID := range docRoomsLeft {
h.BroadcastRoom("doc:"+docID, "CRDT_AWARENESS_LEAVE", map[string]interface{}{
"docId": docID,
"userId": c.userID,
}, nil)
}
}
// Document Locking Methods
@@ -329,6 +369,75 @@ func (h *Hub) BroadcastListEvent(listID, event string, payload interface{}) {
h.Broadcast("", listID, event, payload)
}
func (h *Hub) JoinRoom(c *Client, roomID string) {
h.mu.Lock()
defer h.mu.Unlock()
if h.rooms[roomID] == nil {
h.rooms[roomID] = make(map[*Client]bool)
}
h.rooms[roomID][c] = true
if c.subscribedRooms == nil {
c.subscribedRooms = make(map[string]bool)
}
c.subscribedRooms[roomID] = true
}
func (h *Hub) LeaveRoom(c *Client, roomID string) {
h.mu.Lock()
defer h.mu.Unlock()
if room, ok := h.rooms[roomID]; ok {
delete(room, c)
if len(room) == 0 {
delete(h.rooms, roomID)
}
}
if c.subscribedRooms != nil {
delete(c.subscribedRooms, roomID)
}
}
func (h *Hub) BroadcastRoom(roomID, event string, payload interface{}, excludeClient *Client) {
msg := domain.WSMessage{
Event: event,
Payload: payload,
Timestamp: domain.NowISO(),
}
data, err := json.Marshal(msg)
if err != nil {
return
}
h.mu.RLock()
defer h.mu.RUnlock()
if room, ok := h.rooms[roomID]; ok {
for client := range room {
if excludeClient != nil && client == excludeClient {
continue
}
select {
case client.send <- data:
default:
}
}
}
}
func (c *Client) SendEvent(event string, payload interface{}) {
msg := domain.WSMessage{
Event: event,
Payload: payload,
Timestamp: domain.NowISO(),
}
data, err := json.Marshal(msg)
if err == nil {
select {
case c.send <- data:
default:
}
}
}
func (h *Hub) ServeWS(w http.ResponseWriter, r *http.Request, roomID, userID string) {
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
@@ -355,7 +464,7 @@ func (c *Client) readPump() {
_ = c.conn.Close()
}()
c.conn.SetReadLimit(16384)
c.conn.SetReadLimit(4 * 1024 * 1024) // 4MB for large document CRDT updates
_ = c.conn.SetReadDeadline(time.Now().Add(60 * time.Second))
c.conn.SetPongHandler(func(string) error {
_ = c.conn.SetReadDeadline(time.Now().Add(60 * time.Second))
@@ -368,7 +477,7 @@ func (c *Client) readPump() {
break
}
// Handle client-initiated lock and unlock messages
// Handle client-initiated lock, CRDT, and presence messages
var incoming struct {
Event string `json:"event"`
DocID string `json:"docId"`
@@ -396,6 +505,61 @@ func (c *Client) readPump() {
if c.hub.ReleaseDocLock(docID, c.userID, false) {
c.hub.BroadcastDocEvent(docID, "DOC_UNLOCKED", map[string]string{"docId": docID, "doc_id": docID})
}
case "CRDT_JOIN":
c.hub.JoinRoom(c, "doc:"+docID)
if c.hub.crdtStore != nil {
stateBlob, updates, err := c.hub.crdtStore.GetDocumentCRDTState(docID)
if err == nil && (len(stateBlob) > 0 || len(updates) > 0) {
var b64Updates []string
for _, u := range updates {
b64Updates = append(b64Updates, base64.StdEncoding.EncodeToString(u))
}
b64State := ""
if len(stateBlob) > 0 {
b64State = base64.StdEncoding.EncodeToString(stateBlob)
}
c.SendEvent("CRDT_INIT", map[string]interface{}{
"docId": docID,
"state": b64State,
"updates": b64Updates,
})
}
}
case "CRDT_UPDATE":
var p struct {
Update string `json:"update"`
}
_ = json.Unmarshal(incoming.Payload, &p)
if p.Update != "" {
c.hub.BroadcastRoom("doc:"+docID, "CRDT_UPDATE", map[string]interface{}{
"docId": docID,
"update": p.Update,
"fromUserId": c.userID,
}, c)
if c.hub.crdtStore != nil {
if raw, err := base64.StdEncoding.DecodeString(p.Update); err == nil && len(raw) > 0 {
_ = c.hub.crdtStore.AppendDocumentCRDTUpdate(docID, raw, c.userID)
}
}
}
case "CRDT_COMPACT":
var p struct {
State string `json:"state"`
}
_ = json.Unmarshal(incoming.Payload, &p)
if p.State != "" && c.hub.crdtStore != nil {
if raw, err := base64.StdEncoding.DecodeString(p.State); err == nil && len(raw) > 0 {
_ = c.hub.crdtStore.SaveDocumentCRDTState(docID, raw, c.userID)
}
}
case "CRDT_AWARENESS":
c.hub.BroadcastRoom("doc:"+docID, "CRDT_AWARENESS", incoming.Payload, c)
case "CRDT_LEAVE":
c.hub.LeaveRoom(c, "doc:"+docID)
c.hub.BroadcastRoom("doc:"+docID, "CRDT_AWARENESS_LEAVE", map[string]interface{}{
"docId": docID,
"userId": c.userID,
}, c)
}
}
}