feat(voice): all microphone buttons use the server's speech recognition
MindSift CI & Docker Release / Build, Test in Container & Push (push) Successful in 1m56s
MindSift CI & Docker Release / Build, Test in Container & Push (push) Successful in 1m56s
- New useVoiceInput hook (lib/voice.ts): "server" records with MediaRecorder and sends to /voice/transcribe, "browser" keeps the live recognition, "off" or missing browser support hides the microphone. - Quick dump bar, mobile capture sheet and hold-to-record use it; live waves from the real input level (VoiceWave) and "Wird erkannt…". - Recordings stop after STT_MAX_SECONDS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7801cd13b9
commit
7fa21e485b
9 files changed
+753
-248
No files matched your search
@@ -10,6 +10,9 @@ Das Format folgt [Keep a Changelog](https://keepachangelog.com/de/1.1.0/), die V
|
|||||||
- Eigene Spracherkennung (optional): Mit `STT_URL` schickt MindSift Aufnahmen an einen selbst betriebenen Whisper-Server (OpenAI-kompatibel, z. B. speaches) statt an Google oder Microsoft. Einstellung nur per Umgebungsvariablen (`STT_*`) inklusive Grenzwerten; `docker-compose.yml` bringt den Dienst `whisper` als Profil mit, das Modell lädt MindSift beim ersten Start selbst. Administration → Server & Isolation zeigt den aktiven Modus und bietet eine Testaufnahme. In der Demo bleibt es bei der Browser-Erkennung.
|
- Eigene Spracherkennung (optional): Mit `STT_URL` schickt MindSift Aufnahmen an einen selbst betriebenen Whisper-Server (OpenAI-kompatibel, z. B. speaches) statt an Google oder Microsoft. Einstellung nur per Umgebungsvariablen (`STT_*`) inklusive Grenzwerten; `docker-compose.yml` bringt den Dienst `whisper` als Profil mit, das Modell lädt MindSift beim ersten Start selbst. Administration → Server & Isolation zeigt den aktiven Modus und bietet eine Testaufnahme. In der Demo bleibt es bei der Browser-Erkennung.
|
||||||
- API: `POST /voice/transcribe`, `GET /admin/voice`; `/auth/status` meldet `voice.mode`.
|
- API: `POST /voice/transcribe`, `GET /admin/voice`; `/auth/status` meldet `voice.mode`.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Spracheingabe in Quick Dump, im mobilen Erfassen-Sheet und beim Halten des +-Knopfs folgt der Server-Einstellung: mit eigenem Whisper-Server funktioniert sie in allen Browsern (auch Firefox), mit Live-Wellen während der Aufnahme und „Wird erkannt…“ danach. Ohne Unterstützung im Browser oder mit `STT_DISABLED=true` ist das Mikrofon ausgeblendet statt einer Fehlermeldung.
|
||||||
|
|
||||||
## [0.6.1] - 2026-10-29
|
## [0.6.1] - 2026-10-29
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|||||||
@@ -92,7 +92,7 @@ Hinweis: `DATABASE_URL` mit `postgres…` führt dazu, dass der Standardpfad in
|
|||||||
|
|
||||||
## Spracherkennung
|
## Spracherkennung
|
||||||
|
|
||||||
Nur über Umgebungsvariablen (`STT_*`, siehe oben), nicht in der Oberfläche; Administration → Server & Isolation zeigt den aktiven Modus, den Zustand des Whisper-Servers und bietet eine Testaufnahme. Im Server-Modus nimmt der Browser per `MediaRecorder` auf und schickt die Aufnahme an `POST /voice/transcribe`; MindSift reicht sie an `STT_URL/v1/audio/transcriptions` weiter (`model`, `language`) und gibt nur den Text zurück. Nichts wird gespeichert oder protokolliert außer Fehlern (`internal/api/voice.go`). Beim Start wartet MindSift auf den Whisper-Server und stößt den Download des Modells an, falls es fehlt (`POST /v1/models/<id>` bei speaches).
|
Nur über Umgebungsvariablen (`STT_*`, siehe oben), nicht in der Oberfläche; Administration → Server & Isolation zeigt den aktiven Modus, den Zustand des Whisper-Servers und bietet eine Testaufnahme. Alle Mikrofon-Knöpfe (Quick-Dump-Leiste, mobiles Erfassen-Sheet, Halten des +-Knopfs) nutzen denselben Hook `useVoiceInput` (`frontend/src/lib/voice.ts`, `useVoiceInput.ts`): `server` nimmt auf und zeigt „Wird erkannt…“, `browser` liefert Text schon beim Sprechen, `off` oder ein Browser ohne die nötige Funktion blendet das Mikrofon aus. Im Server-Modus zeigen Wellen den echten Mikrofonpegel (`VoiceWave`), die Aufnahme endet spätestens nach `STT_MAX_SECONDS`. Der Browser nimmt per `MediaRecorder` auf (WebM/Opus, in Safari MP4/AAC) und schickt die Aufnahme an `POST /voice/transcribe`; MindSift reicht sie an `STT_URL/v1/audio/transcriptions` weiter (`model`, `language`) und gibt nur den Text zurück. Nichts wird gespeichert oder protokolliert außer Fehlern (`internal/api/voice.go`). Beim Start wartet MindSift auf den Whisper-Server und stößt den Download des Modells an, falls es fehlt (`POST /v1/models/<id>` bei speaches).
|
||||||
|
|
||||||
Docker Compose bringt den optionalen Dienst `whisper` (speaches 0.8.3, CPU, Profil `whisper`) mit; das Modell liegt in `./whisper` (Ordner gehört UID 1000). Gemessen auf einem Desktop mit 16 Kernen (int8) für einen Satz von 4 s: `Systran/faster-whisper-small` 1,6 s, `deepdml/faster-whisper-large-v3-turbo-ct2` 8 s bei besserer Erkennung (RAM ca. 1 bzw. 2 GB). Auf geteilten vCPUs ist mit dem Mehrfachen zu rechnen.
|
Docker Compose bringt den optionalen Dienst `whisper` (speaches 0.8.3, CPU, Profil `whisper`) mit; das Modell liegt in `./whisper` (Ordner gehört UID 1000). Gemessen auf einem Desktop mit 16 Kernen (int8) für einen Satz von 4 s: `Systran/faster-whisper-small` 1,6 s, `deepdml/faster-whisper-large-v3-turbo-ct2` 8 s bei besserer Erkennung (RAM ca. 1 bzw. 2 GB). Auf geteilten vCPUs ist mit dem Mehrfachen zu rechnen.
|
||||||
|
|
||||||
|
|||||||
+65
-52
@@ -23,6 +23,7 @@ import {
|
|||||||
Flame,
|
Flame,
|
||||||
Mic,
|
Mic,
|
||||||
MicOff,
|
MicOff,
|
||||||
|
Loader2,
|
||||||
Search,
|
Search,
|
||||||
} from 'lucide-react';
|
} from 'lucide-react';
|
||||||
import { AuthStatus, DashboardStats, PriorityLevel, SharedList, Thought, User, UserTheme } from './lib/types';
|
import { AuthStatus, DashboardStats, PriorityLevel, SharedList, Thought, User, UserTheme } from './lib/types';
|
||||||
@@ -35,6 +36,8 @@ import {
|
|||||||
disconnectWebSocket,
|
disconnectWebSocket,
|
||||||
isSharingDisabled,
|
isSharingDisabled,
|
||||||
} from './lib/api';
|
} from './lib/api';
|
||||||
|
import { useVoiceInput } from './lib/useVoiceInput';
|
||||||
|
import { VoiceWave } from './components/VoiceWave';
|
||||||
import {
|
import {
|
||||||
AVAILABLE_THEMES,
|
AVAILABLE_THEMES,
|
||||||
customThemeOption,
|
customThemeOption,
|
||||||
@@ -175,49 +178,31 @@ export const App: React.FC = () => {
|
|||||||
const [mobileTriageContent, setMobileTriageContent] = useState('');
|
const [mobileTriageContent, setMobileTriageContent] = useState('');
|
||||||
const [mobileTriagePriority, setMobileTriagePriority] = useState<PriorityLevel>('normal');
|
const [mobileTriagePriority, setMobileTriagePriority] = useState<PriorityLevel>('normal');
|
||||||
const [mobileTriageDueDate, setMobileTriageDueDate] = useState('');
|
const [mobileTriageDueDate, setMobileTriageDueDate] = useState('');
|
||||||
const [mobileVoiceActive, setMobileVoiceActive] = useState(false);
|
|
||||||
const [isVoiceCaptured, setIsVoiceCaptured] = useState(false);
|
const [isVoiceCaptured, setIsVoiceCaptured] = useState(false);
|
||||||
const mobileRecognitionRef = useRef<any>(null);
|
const [mobileVoiceError, setMobileVoiceError] = useState<string | null>(null);
|
||||||
const mobileBaseVoiceRef = useRef('');
|
const mobileBaseVoiceRef = useRef('');
|
||||||
|
// Voice input of the mobile capture sheet: own Whisper server or the browser (lib/voice.ts)
|
||||||
|
const mobileVoice = useVoiceInput({
|
||||||
|
onText: (text) => {
|
||||||
|
if (!text) return;
|
||||||
|
const base = mobileBaseVoiceRef.current;
|
||||||
|
setMobileTriageContent(base ? `${base} ${text}` : text);
|
||||||
|
setIsVoiceCaptured(true);
|
||||||
|
},
|
||||||
|
onError: (message) => {
|
||||||
|
setMobileVoiceError(message);
|
||||||
|
window.setTimeout(() => setMobileVoiceError(null), 4000);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const mobileVoiceActive = mobileVoice.state === 'listening';
|
||||||
|
|
||||||
const toggleMobileVoice = () => {
|
const toggleMobileVoice = () => {
|
||||||
const SpeechRecognition =
|
if (mobileVoice.state === 'idle') {
|
||||||
(window as any).SpeechRecognition || (window as any).webkitSpeechRecognition;
|
|
||||||
if (!SpeechRecognition) {
|
|
||||||
alert('Spracheingabe wird von diesem Browser leider nicht unterstützt.');
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (mobileVoiceActive && mobileRecognitionRef.current) {
|
|
||||||
mobileRecognitionRef.current.stop();
|
|
||||||
setMobileVoiceActive(false);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
mobileBaseVoiceRef.current = mobileTriageContent.replace(/\s*\[…\]$/, '').trim();
|
mobileBaseVoiceRef.current = mobileTriageContent.replace(/\s*\[…\]$/, '').trim();
|
||||||
const recognition = new SpeechRecognition();
|
setMobileVoiceError(null);
|
||||||
recognition.lang = 'de-DE';
|
triggerHaptic(20);
|
||||||
recognition.continuous = false;
|
|
||||||
recognition.interimResults = true;
|
|
||||||
recognition.onstart = () => {
|
|
||||||
setMobileVoiceActive(true);
|
|
||||||
triggerHaptic(20);
|
|
||||||
};
|
|
||||||
recognition.onresult = (event: any) => {
|
|
||||||
let transcript = '';
|
|
||||||
for (let i = 0; i < event.results.length; i++) {
|
|
||||||
transcript += event.results[i][0].transcript;
|
|
||||||
}
|
|
||||||
const base = mobileBaseVoiceRef.current;
|
|
||||||
setMobileTriageContent(base ? `${base} ${transcript}` : transcript);
|
|
||||||
setIsVoiceCaptured(true);
|
|
||||||
};
|
|
||||||
recognition.onerror = () => setMobileVoiceActive(false);
|
|
||||||
recognition.onend = () => setMobileVoiceActive(false);
|
|
||||||
mobileRecognitionRef.current = recognition;
|
|
||||||
recognition.start();
|
|
||||||
} catch {
|
|
||||||
setMobileVoiceActive(false);
|
|
||||||
}
|
}
|
||||||
|
mobileVoice.toggle();
|
||||||
};
|
};
|
||||||
|
|
||||||
const handleMobileCreateTriage = async (e: React.FormEvent) => {
|
const handleMobileCreateTriage = async (e: React.FormEvent) => {
|
||||||
@@ -249,7 +234,7 @@ export const App: React.FC = () => {
|
|||||||
setMobileTriagePriority('normal');
|
setMobileTriagePriority('normal');
|
||||||
setMobileTriageDueDate('');
|
setMobileTriageDueDate('');
|
||||||
setIsVoiceCaptured(false);
|
setIsVoiceCaptured(false);
|
||||||
setMobileVoiceActive(false);
|
if (mobileVoice.state !== 'idle') mobileVoice.cancel();
|
||||||
setShowMobileCapture(true);
|
setShowMobileCapture(true);
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -1341,21 +1326,49 @@ export const App: React.FC = () => {
|
|||||||
autoFocus
|
autoFocus
|
||||||
value={mobileTriageContent}
|
value={mobileTriageContent}
|
||||||
onChange={(e) => setMobileTriageContent(e.target.value)}
|
onChange={(e) => setMobileTriageContent(e.target.value)}
|
||||||
placeholder="Was geht dir durch den Kopf?…"
|
placeholder={
|
||||||
className="w-full input-field px-3 py-2 pr-10 text-sm"
|
mobileVoiceError
|
||||||
|
? mobileVoiceError
|
||||||
|
: mobileVoiceActive
|
||||||
|
? 'Ich höre zu…'
|
||||||
|
: mobileVoice.state === 'transcribing'
|
||||||
|
? 'Wird erkannt…'
|
||||||
|
: 'Was geht dir durch den Kopf?…'
|
||||||
|
}
|
||||||
|
className={
|
||||||
|
mobileVoiceActive && mobileVoice.mode === 'server'
|
||||||
|
? 'w-full input-field px-3 py-2 pr-24 text-sm'
|
||||||
|
: 'w-full input-field px-3 py-2 pr-10 text-sm'
|
||||||
|
}
|
||||||
/>
|
/>
|
||||||
<button
|
{mobileVoice.available && (
|
||||||
type="button"
|
<div className="absolute right-1.5 top-1/2 -translate-y-1/2 flex items-center gap-2">
|
||||||
onClick={toggleMobileVoice}
|
{mobileVoiceActive && mobileVoice.mode === 'server' && (
|
||||||
className={`absolute right-1.5 top-1/2 -translate-y-1/2 w-7 h-7 rounded-full flex items-center justify-center transition-colors cursor-pointer ${
|
<VoiceWave getLevel={mobileVoice.level} bars={7} className="h-4 text-accent" />
|
||||||
mobileVoiceActive
|
)}
|
||||||
? 'bg-accent text-on-accent animate-pulse'
|
<button
|
||||||
: 'text-muted hover:text-main hover:bg-panel-elevated'
|
type="button"
|
||||||
}`}
|
onClick={toggleMobileVoice}
|
||||||
title="Spracheingabe"
|
disabled={mobileVoice.state === 'transcribing'}
|
||||||
>
|
className={`w-9 h-9 rounded-full flex items-center justify-center transition-colors cursor-pointer ${
|
||||||
{mobileVoiceActive ? <MicOff className="w-3.5 h-3.5" /> : <Mic className="w-3.5 h-3.5" />}
|
mobileVoiceActive
|
||||||
</button>
|
? mobileVoice.mode === 'server'
|
||||||
|
? 'bg-status-err text-on-accent'
|
||||||
|
: 'bg-accent text-on-accent animate-pulse'
|
||||||
|
: 'text-muted hover:text-main hover:bg-panel-elevated'
|
||||||
|
}`}
|
||||||
|
title={mobileVoiceActive ? 'Aufnahme beenden' : 'Spracheingabe'}
|
||||||
|
>
|
||||||
|
{mobileVoice.state === 'transcribing' ? (
|
||||||
|
<Loader2 className="w-4 h-4 animate-spin" />
|
||||||
|
) : mobileVoiceActive ? (
|
||||||
|
<MicOff className="w-4 h-4" />
|
||||||
|
) : (
|
||||||
|
<Mic className="w-4 h-4" />
|
||||||
|
)}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
import React, { useEffect, useRef } from 'react';
|
||||||
|
|
||||||
|
interface VoiceWaveProps {
|
||||||
|
/** Input level 0..1; null (unknown, e.g. browser recognition) shows a calm idle wave. */
|
||||||
|
getLevel: () => number | null;
|
||||||
|
bars?: number;
|
||||||
|
/** Bar colour and size come from the caller, e.g. "h-4 text-accent". */
|
||||||
|
className?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Live wave reacting to the voice. Heights are set directly on the bars every frame, so React does
|
||||||
|
// not re-render 60 times a second.
|
||||||
|
export const VoiceWave: React.FC<VoiceWaveProps> = ({ getLevel, bars = 5, className = 'h-4 text-accent' }) => {
|
||||||
|
const barRefs = useRef<(HTMLSpanElement | null)[]>([]);
|
||||||
|
const getLevelRef = useRef(getLevel);
|
||||||
|
getLevelRef.current = getLevel;
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
const history: number[] = new Array(bars).fill(0);
|
||||||
|
let frame = 0;
|
||||||
|
let raf = 0;
|
||||||
|
const reduceMotion = window.matchMedia?.('(prefers-reduced-motion: reduce)').matches;
|
||||||
|
const tick = () => {
|
||||||
|
frame++;
|
||||||
|
const measured = getLevelRef.current();
|
||||||
|
const level = measured ?? 0.25 + 0.2 * Math.sin(frame / 9);
|
||||||
|
// Newest level enters in the middle and travels outwards
|
||||||
|
if (frame % 3 === 0) {
|
||||||
|
history.pop();
|
||||||
|
history.unshift(level);
|
||||||
|
}
|
||||||
|
const mid = Math.floor(bars / 2);
|
||||||
|
barRefs.current.forEach((el, i) => {
|
||||||
|
if (!el) return;
|
||||||
|
const v = reduceMotion ? 0.5 : history[Math.abs(i - mid)] ?? 0;
|
||||||
|
el.style.transform = `scaleY(${Math.max(0.15, Math.min(1, v * 1.4))})`;
|
||||||
|
});
|
||||||
|
raf = requestAnimationFrame(tick);
|
||||||
|
};
|
||||||
|
raf = requestAnimationFrame(tick);
|
||||||
|
return () => cancelAnimationFrame(raf);
|
||||||
|
}, [bars]);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<span className={`inline-flex items-center gap-[3px] ${className}`} aria-hidden="true">
|
||||||
|
{Array.from({ length: bars }, (_, i) => (
|
||||||
|
<span
|
||||||
|
key={i}
|
||||||
|
ref={(el) => {
|
||||||
|
barRefs.current[i] = el;
|
||||||
|
}}
|
||||||
|
className="w-[3px] h-full rounded-full bg-current origin-center transition-transform duration-75"
|
||||||
|
style={{ transform: 'scaleY(0.15)' }}
|
||||||
|
/>
|
||||||
|
))}
|
||||||
|
</span>
|
||||||
|
);
|
||||||
|
};
|
||||||
@@ -1,6 +1,8 @@
|
|||||||
import React, { useEffect, useRef, useState } from 'react';
|
import React, { useEffect, useRef, useState } from 'react';
|
||||||
import { Mic, Plus } from 'lucide-react';
|
import { Mic, Plus } from 'lucide-react';
|
||||||
import { triggerHaptic } from '../../lib/api';
|
import { triggerHaptic } from '../../lib/api';
|
||||||
|
import { useVoiceInput } from '../../lib/useVoiceInput';
|
||||||
|
import { VoiceWave } from '../../components/VoiceWave';
|
||||||
|
|
||||||
export interface HoldToRecordFabProps {
|
export interface HoldToRecordFabProps {
|
||||||
onTap: () => void;
|
onTap: () => void;
|
||||||
@@ -24,14 +26,10 @@ export const HoldToRecordFab: React.FC<HoldToRecordFabProps> = ({ onTap, onTrans
|
|||||||
const didEnterHoldRef = useRef(false);
|
const didEnterHoldRef = useRef(false);
|
||||||
const fallbackHandledRef = useRef(false);
|
const fallbackHandledRef = useRef(false);
|
||||||
const isCancelZoneRef = useRef(false);
|
const isCancelZoneRef = useRef(false);
|
||||||
const abortedRef = useRef(false);
|
|
||||||
const isStoppingRef = useRef(false);
|
|
||||||
const hasEndedRef = useRef(false);
|
|
||||||
const finalizedRef = useRef(false);
|
const finalizedRef = useRef(false);
|
||||||
const startYRef = useRef(0);
|
const startYRef = useRef(0);
|
||||||
const transcriptRef = useRef('');
|
const transcriptRef = useRef('');
|
||||||
|
|
||||||
const recognitionRef = useRef<any>(null);
|
|
||||||
const holdTimerRef = useRef<number | null>(null);
|
const holdTimerRef = useRef<number | null>(null);
|
||||||
const stopTimerRef = useRef<number | null>(null);
|
const stopTimerRef = useRef<number | null>(null);
|
||||||
const toastTimerRef = useRef<number | null>(null);
|
const toastTimerRef = useRef<number | null>(null);
|
||||||
@@ -50,17 +48,66 @@ export const HoldToRecordFab: React.FC<HoldToRecordFabProps> = ({ onTap, onTrans
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
const showToast = (msg: string) => {
|
const showToast = (msg: string, sticky = false) => {
|
||||||
if (toastTimerRef.current !== null) {
|
if (toastTimerRef.current !== null) {
|
||||||
window.clearTimeout(toastTimerRef.current);
|
window.clearTimeout(toastTimerRef.current);
|
||||||
|
toastTimerRef.current = null;
|
||||||
}
|
}
|
||||||
setToastMessage(msg);
|
setToastMessage(msg);
|
||||||
|
if (sticky) return;
|
||||||
toastTimerRef.current = window.setTimeout(() => {
|
toastTimerRef.current = window.setTimeout(() => {
|
||||||
setToastMessage(null);
|
setToastMessage(null);
|
||||||
toastTimerRef.current = null;
|
toastTimerRef.current = null;
|
||||||
}, TOAST_DURATION_MS);
|
}, TOAST_DURATION_MS);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const finalizeRecording = async (text: string) => {
|
||||||
|
if (finalizedRef.current) return;
|
||||||
|
finalizedRef.current = true;
|
||||||
|
clearStopTimer();
|
||||||
|
setLiveTranscript('');
|
||||||
|
const trimmed = text.trim();
|
||||||
|
if (!trimmed) {
|
||||||
|
showToast('Nichts erkannt');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
await onTranscript(trimmed);
|
||||||
|
triggerHaptic(25);
|
||||||
|
showToast('Im Quick Dump gespeichert');
|
||||||
|
} catch {
|
||||||
|
showToast('Fehler beim Speichern');
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Own Whisper server (text after release) or the browser's recognition (text while speaking)
|
||||||
|
const voice = useVoiceInput({
|
||||||
|
continuous: true,
|
||||||
|
onText: (text, final) => {
|
||||||
|
transcriptRef.current = text;
|
||||||
|
if (!final) {
|
||||||
|
setLiveTranscript(text);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
void finalizeRecording(text);
|
||||||
|
},
|
||||||
|
onError: (message) => {
|
||||||
|
clearStopTimer();
|
||||||
|
setIsRecording(false);
|
||||||
|
setIsCancelZone(false);
|
||||||
|
setLiveTranscript('');
|
||||||
|
if (didEnterHoldRef.current && isPressedRef.current && !fallbackHandledRef.current) {
|
||||||
|
// Recording could not start: open the normal capture sheet instead
|
||||||
|
fallbackHandledRef.current = true;
|
||||||
|
showToast(message);
|
||||||
|
onTap();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
finalizedRef.current = true;
|
||||||
|
showToast(message);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
return () => {
|
return () => {
|
||||||
clearHoldTimer();
|
clearHoldTimer();
|
||||||
@@ -68,112 +115,28 @@ export const HoldToRecordFab: React.FC<HoldToRecordFabProps> = ({ onTap, onTrans
|
|||||||
if (toastTimerRef.current !== null) {
|
if (toastTimerRef.current !== null) {
|
||||||
window.clearTimeout(toastTimerRef.current);
|
window.clearTimeout(toastTimerRef.current);
|
||||||
}
|
}
|
||||||
if (recognitionRef.current) {
|
|
||||||
try {
|
|
||||||
recognitionRef.current.abort?.();
|
|
||||||
} catch {
|
|
||||||
// ignore cleanup errors
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
};
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
const finalizeRecording = async () => {
|
useEffect(() => {
|
||||||
if (finalizedRef.current) return;
|
if (voice.state === 'transcribing') showToast('Wird erkannt…', true);
|
||||||
finalizedRef.current = true;
|
}, [voice.state]);
|
||||||
clearStopTimer();
|
|
||||||
recognitionRef.current = null;
|
|
||||||
|
|
||||||
if (abortedRef.current) return;
|
|
||||||
|
|
||||||
const text = transcriptRef.current.trim();
|
|
||||||
setLiveTranscript('');
|
|
||||||
if (!text) {
|
|
||||||
showToast('Nichts erkannt');
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
|
||||||
await onTranscript(text);
|
|
||||||
triggerHaptic(25);
|
|
||||||
showToast('Im Quick Dump gespeichert');
|
|
||||||
} catch {
|
|
||||||
showToast('Fehler beim Speichern');
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
const startVoiceRecording = () => {
|
const startVoiceRecording = () => {
|
||||||
const SpeechRecognition =
|
if (voice.mode === 'off' || voice.mode === 'unsupported') {
|
||||||
(window as any).SpeechRecognition || (window as any).webkitSpeechRecognition;
|
|
||||||
|
|
||||||
if (!SpeechRecognition) {
|
|
||||||
fallbackHandledRef.current = true;
|
fallbackHandledRef.current = true;
|
||||||
showToast('Spracheingabe wird von diesem Browser nicht unterstützt');
|
if (voice.mode === 'unsupported') showToast('Spracheingabe wird von diesem Browser nicht unterstützt');
|
||||||
onTap();
|
onTap();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
transcriptRef.current = '';
|
||||||
try {
|
finalizedRef.current = false;
|
||||||
const recognition = new SpeechRecognition();
|
isCancelZoneRef.current = false;
|
||||||
recognition.lang = 'de-DE';
|
setLiveTranscript('');
|
||||||
recognition.interimResults = true;
|
setIsCancelZone(false);
|
||||||
recognition.continuous = true;
|
setIsRecording(true);
|
||||||
|
triggerHaptic([15, 60, 25]);
|
||||||
transcriptRef.current = '';
|
void voice.start();
|
||||||
abortedRef.current = false;
|
|
||||||
isStoppingRef.current = false;
|
|
||||||
hasEndedRef.current = false;
|
|
||||||
finalizedRef.current = false;
|
|
||||||
isCancelZoneRef.current = false;
|
|
||||||
|
|
||||||
recognition.onresult = (event: any) => {
|
|
||||||
let combined = '';
|
|
||||||
for (let i = 0; i < event.results.length; i++) {
|
|
||||||
combined += event.results[i][0].transcript;
|
|
||||||
}
|
|
||||||
transcriptRef.current = combined;
|
|
||||||
setLiveTranscript(combined);
|
|
||||||
};
|
|
||||||
|
|
||||||
recognition.onerror = (event: any) => {
|
|
||||||
const errCode = event?.error;
|
|
||||||
if (errCode === 'not-allowed' || errCode === 'service-not-allowed') {
|
|
||||||
abortedRef.current = true;
|
|
||||||
clearStopTimer();
|
|
||||||
setIsRecording(false);
|
|
||||||
setIsCancelZone(false);
|
|
||||||
setLiveTranscript('');
|
|
||||||
if (!fallbackHandledRef.current) {
|
|
||||||
fallbackHandledRef.current = true;
|
|
||||||
showToast('Mikrofon nicht freigegeben');
|
|
||||||
onTap();
|
|
||||||
}
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (isStoppingRef.current) {
|
|
||||||
void finalizeRecording();
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
recognition.onend = () => {
|
|
||||||
hasEndedRef.current = true;
|
|
||||||
if (isStoppingRef.current) {
|
|
||||||
void finalizeRecording();
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
recognitionRef.current = recognition;
|
|
||||||
setLiveTranscript('');
|
|
||||||
setIsCancelZone(false);
|
|
||||||
setIsRecording(true);
|
|
||||||
triggerHaptic([15, 60, 25]);
|
|
||||||
recognition.start();
|
|
||||||
} catch {
|
|
||||||
fallbackHandledRef.current = true;
|
|
||||||
setIsRecording(false);
|
|
||||||
showToast('Spracheingabe wird von diesem Browser nicht unterstützt');
|
|
||||||
onTap();
|
|
||||||
}
|
|
||||||
};
|
};
|
||||||
|
|
||||||
const handlePointerDown = (e: React.PointerEvent<HTMLButtonElement>) => {
|
const handlePointerDown = (e: React.PointerEvent<HTMLButtonElement>) => {
|
||||||
@@ -240,34 +203,23 @@ export const HoldToRecordFab: React.FC<HoldToRecordFabProps> = ({ onTap, onTrans
|
|||||||
setIsCancelZone(false);
|
setIsCancelZone(false);
|
||||||
|
|
||||||
if (isCancelZoneRef.current) {
|
if (isCancelZoneRef.current) {
|
||||||
abortedRef.current = true;
|
finalizedRef.current = true;
|
||||||
clearStopTimer();
|
clearStopTimer();
|
||||||
setLiveTranscript('');
|
setLiveTranscript('');
|
||||||
triggerHaptic(15);
|
triggerHaptic(15);
|
||||||
try {
|
voice.cancel();
|
||||||
recognitionRef.current?.abort?.();
|
|
||||||
} catch {
|
|
||||||
// ignore
|
|
||||||
}
|
|
||||||
recognitionRef.current = null;
|
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
isStoppingRef.current = true;
|
// The browser's recognition sometimes never reports its end: save what was heard after a moment.
|
||||||
if (hasEndedRef.current || !recognitionRef.current) {
|
// The own server answers when it is done (with its own time limit).
|
||||||
void finalizeRecording();
|
if (voice.mode !== 'server') {
|
||||||
return;
|
stopTimerRef.current = window.setTimeout(() => {
|
||||||
}
|
voice.cancel();
|
||||||
|
void finalizeRecording(transcriptRef.current);
|
||||||
stopTimerRef.current = window.setTimeout(() => {
|
}, STOP_TIMEOUT_MS);
|
||||||
void finalizeRecording();
|
|
||||||
}, STOP_TIMEOUT_MS);
|
|
||||||
|
|
||||||
try {
|
|
||||||
recognitionRef.current.stop();
|
|
||||||
} catch {
|
|
||||||
void finalizeRecording();
|
|
||||||
}
|
}
|
||||||
|
void voice.stop();
|
||||||
};
|
};
|
||||||
|
|
||||||
const handlePointerCancel = (e: React.PointerEvent<HTMLButtonElement>) => {
|
const handlePointerCancel = (e: React.PointerEvent<HTMLButtonElement>) => {
|
||||||
@@ -283,16 +235,11 @@ export const HoldToRecordFab: React.FC<HoldToRecordFabProps> = ({ onTap, onTrans
|
|||||||
clearStopTimer();
|
clearStopTimer();
|
||||||
|
|
||||||
if (didEnterHoldRef.current && !fallbackHandledRef.current) {
|
if (didEnterHoldRef.current && !fallbackHandledRef.current) {
|
||||||
abortedRef.current = true;
|
finalizedRef.current = true;
|
||||||
setIsRecording(false);
|
setIsRecording(false);
|
||||||
setIsCancelZone(false);
|
setIsCancelZone(false);
|
||||||
setLiveTranscript('');
|
setLiveTranscript('');
|
||||||
try {
|
voice.cancel();
|
||||||
recognitionRef.current?.abort?.();
|
|
||||||
} catch {
|
|
||||||
// ignore
|
|
||||||
}
|
|
||||||
recognitionRef.current = null;
|
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -313,7 +260,8 @@ export const HoldToRecordFab: React.FC<HoldToRecordFabProps> = ({ onTap, onTrans
|
|||||||
<div className="text-xs font-semibold">🗑️ Zum Abbrechen loslassen</div>
|
<div className="text-xs font-semibold">🗑️ Zum Abbrechen loslassen</div>
|
||||||
) : (
|
) : (
|
||||||
<div className="space-y-1">
|
<div className="space-y-1">
|
||||||
<div className="text-xs font-semibold text-main">
|
<div className="text-xs font-semibold text-main flex items-center justify-center gap-2">
|
||||||
|
{voice.mode === 'server' && <VoiceWave getLevel={voice.level} bars={7} className="h-4 text-accent" />}
|
||||||
Ich höre zu… (loslassen speichert)
|
Ich höre zu… (loslassen speichert)
|
||||||
</div>
|
</div>
|
||||||
{displayedTranscript ? (
|
{displayedTranscript ? (
|
||||||
|
|||||||
@@ -13,9 +13,12 @@ import {
|
|||||||
ChevronDown,
|
ChevronDown,
|
||||||
AlertTriangle,
|
AlertTriangle,
|
||||||
Flame,
|
Flame,
|
||||||
|
Loader2,
|
||||||
} from 'lucide-react';
|
} from 'lucide-react';
|
||||||
import { PriorityLevel, SharedList, Thought } from '../../lib/types';
|
import { PriorityLevel, SharedList, Thought } from '../../lib/types';
|
||||||
import { api, triggerHaptic } from '../../lib/api';
|
import { api, triggerHaptic } from '../../lib/api';
|
||||||
|
import { useVoiceInput } from '../../lib/useVoiceInput';
|
||||||
|
import { VoiceWave } from '../../components/VoiceWave';
|
||||||
|
|
||||||
export type CaptureTarget = 'inbox' | 'task' | 'list_item' | 'document';
|
export type CaptureTarget = 'inbox' | 'task' | 'list_item' | 'document';
|
||||||
|
|
||||||
@@ -46,17 +49,31 @@ export const InstantCaptureBar: React.FC<InstantCaptureBarProps> = ({
|
|||||||
const [target, setTarget] = useState<CaptureTarget>(fixedTarget || 'inbox');
|
const [target, setTarget] = useState<CaptureTarget>(fixedTarget || 'inbox');
|
||||||
const [targetListId, setTargetListId] = useState<string>('');
|
const [targetListId, setTargetListId] = useState<string>('');
|
||||||
const [showOptions, setShowOptions] = useState(defaultShowOptions || hideTargetPicker);
|
const [showOptions, setShowOptions] = useState(defaultShowOptions || hideTargetPicker);
|
||||||
const [isListening, setIsListening] = useState(false);
|
const [voiceError, setVoiceError] = useState<string | null>(null);
|
||||||
const [isSubmitting, setIsSubmitting] = useState(false);
|
const [isSubmitting, setIsSubmitting] = useState(false);
|
||||||
const [justSavedLabel, setJustSavedLabel] = useState<string | null>(null);
|
const [justSavedLabel, setJustSavedLabel] = useState<string | null>(null);
|
||||||
const [isMultiline, setIsMultiline] = useState(false);
|
const [isMultiline, setIsMultiline] = useState(false);
|
||||||
|
|
||||||
const inputRef = useRef<HTMLTextAreaElement>(null);
|
const inputRef = useRef<HTMLTextAreaElement>(null);
|
||||||
const containerRef = useRef<HTMLFormElement>(null);
|
const containerRef = useRef<HTMLFormElement>(null);
|
||||||
const recognitionRef = useRef<any>(null);
|
|
||||||
const baseVoiceTextRef = useRef('');
|
const baseVoiceTextRef = useRef('');
|
||||||
const hasVoiceInputRef = useRef(false);
|
const hasVoiceInputRef = useRef(false);
|
||||||
|
|
||||||
|
// Voice input: own Whisper server or the browser's recognition (lib/voice.ts)
|
||||||
|
const voice = useVoiceInput({
|
||||||
|
onText: (text, final) => {
|
||||||
|
const base = baseVoiceTextRef.current;
|
||||||
|
if (text) setContent(base ? `${base} ${text}` : text);
|
||||||
|
if (final) inputRef.current?.focus();
|
||||||
|
},
|
||||||
|
onError: (message) => {
|
||||||
|
setVoiceError(message);
|
||||||
|
window.setTimeout(() => setVoiceError(null), 4000);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const isListening = voice.state === 'listening';
|
||||||
|
const isTranscribing = voice.state === 'transcribing';
|
||||||
|
|
||||||
const adjustHeight = (el?: HTMLTextAreaElement | null) => {
|
const adjustHeight = (el?: HTMLTextAreaElement | null) => {
|
||||||
const targetEl = el || inputRef.current;
|
const targetEl = el || inputRef.current;
|
||||||
if (!targetEl) return;
|
if (!targetEl) return;
|
||||||
@@ -120,57 +137,13 @@ export const InstantCaptureBar: React.FC<InstantCaptureBarProps> = ({
|
|||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
const toggleVoiceDictation = () => {
|
const toggleVoiceDictation = () => {
|
||||||
const SpeechRecognition =
|
if (voice.state === 'idle') {
|
||||||
(window as any).SpeechRecognition || (window as any).webkitSpeechRecognition;
|
|
||||||
|
|
||||||
if (!SpeechRecognition) {
|
|
||||||
alert('Spracheingabe wird von diesem Browser nicht direkt unterstützt (bitte Chrome, Edge oder Safari nutzen).');
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (isListening && recognitionRef.current) {
|
|
||||||
recognitionRef.current.stop();
|
|
||||||
setIsListening(false);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
|
||||||
baseVoiceTextRef.current = content.replace(/\s*\[…\]$/, '').trim();
|
baseVoiceTextRef.current = content.replace(/\s*\[…\]$/, '').trim();
|
||||||
hasVoiceInputRef.current = true;
|
hasVoiceInputRef.current = true;
|
||||||
|
setVoiceError(null);
|
||||||
const recognition = new SpeechRecognition();
|
triggerHaptic(20);
|
||||||
recognition.lang = 'de-DE';
|
|
||||||
recognition.continuous = false;
|
|
||||||
recognition.interimResults = true;
|
|
||||||
|
|
||||||
recognition.onstart = () => {
|
|
||||||
setIsListening(true);
|
|
||||||
triggerHaptic(20);
|
|
||||||
};
|
|
||||||
|
|
||||||
recognition.onresult = (event: any) => {
|
|
||||||
let transcript = '';
|
|
||||||
for (let i = 0; i < event.results.length; i++) {
|
|
||||||
transcript += event.results[i][0].transcript;
|
|
||||||
}
|
|
||||||
const base = baseVoiceTextRef.current;
|
|
||||||
setContent(base ? `${base} ${transcript}` : transcript);
|
|
||||||
};
|
|
||||||
|
|
||||||
recognition.onerror = () => {
|
|
||||||
setIsListening(false);
|
|
||||||
};
|
|
||||||
|
|
||||||
recognition.onend = () => {
|
|
||||||
setIsListening(false);
|
|
||||||
inputRef.current?.focus();
|
|
||||||
};
|
|
||||||
|
|
||||||
recognitionRef.current = recognition;
|
|
||||||
recognition.start();
|
|
||||||
} catch {
|
|
||||||
setIsListening(false);
|
|
||||||
}
|
}
|
||||||
|
voice.toggle();
|
||||||
};
|
};
|
||||||
|
|
||||||
const executeCapture = async (
|
const executeCapture = async (
|
||||||
@@ -187,6 +160,8 @@ export const InstantCaptureBar: React.FC<InstantCaptureBarProps> = ({
|
|||||||
const capturedDueDate = dueDate;
|
const capturedDueDate = dueDate;
|
||||||
const capturedSource = hasVoiceInputRef.current || isListening ? 'voice' : 'manual';
|
const capturedSource = hasVoiceInputRef.current || isListening ? 'voice' : 'manual';
|
||||||
hasVoiceInputRef.current = false;
|
hasVoiceInputRef.current = false;
|
||||||
|
// A recognition still running would refill the field after it was saved
|
||||||
|
if (voice.state !== 'idle') voice.cancel();
|
||||||
|
|
||||||
// Optimistically clear input immediately so the UI feels instant (<10ms) in all browsers
|
// Optimistically clear input immediately so the UI feels instant (<10ms) in all browsers
|
||||||
setContent('');
|
setContent('');
|
||||||
@@ -285,8 +260,12 @@ export const InstantCaptureBar: React.FC<InstantCaptureBarProps> = ({
|
|||||||
}
|
}
|
||||||
}}
|
}}
|
||||||
placeholder={
|
placeholder={
|
||||||
isListening
|
voiceError
|
||||||
|
? voiceError
|
||||||
|
: isListening
|
||||||
? 'Ich höre zu…'
|
? 'Ich höre zu…'
|
||||||
|
: isTranscribing
|
||||||
|
? 'Wird erkannt…'
|
||||||
: hideTargetPicker || fixedTarget === 'inbox'
|
: hideTargetPicker || fixedTarget === 'inbox'
|
||||||
? 'Quick Dump erfassen… (Enter zum Speichern)'
|
? 'Quick Dump erfassen… (Enter zum Speichern)'
|
||||||
: target === 'task'
|
: target === 'task'
|
||||||
@@ -336,19 +315,33 @@ export const InstantCaptureBar: React.FC<InstantCaptureBarProps> = ({
|
|||||||
</button>
|
</button>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* Voice Dictation Button */}
|
{/* Voice Dictation Button (hidden when voice input is off or not supported) */}
|
||||||
<button
|
{voice.available && (
|
||||||
type="button"
|
<>
|
||||||
onClick={toggleVoiceDictation}
|
{isListening && voice.mode === 'server' && (
|
||||||
title="Spracheingabe"
|
<VoiceWave getLevel={voice.level} className="h-3.5 text-accent" />
|
||||||
className={`w-6 h-6 capture-pill rounded-full flex items-center justify-center transition-colors ${
|
)}
|
||||||
isListening
|
<button
|
||||||
? 'bg-accent-dim text-accent'
|
type="button"
|
||||||
: 'text-muted hover:text-main hover:bg-panel-elevated'
|
onClick={toggleVoiceDictation}
|
||||||
}`}
|
disabled={isTranscribing}
|
||||||
>
|
title={isListening ? 'Aufnahme beenden' : isTranscribing ? 'Wird erkannt…' : 'Spracheingabe'}
|
||||||
{isListening ? <MicOff className="w-3 h-3" /> : <Mic className="w-3 h-3" />}
|
className={`w-6 h-6 capture-pill rounded-full flex items-center justify-center transition-colors ${
|
||||||
</button>
|
isListening || isTranscribing
|
||||||
|
? 'bg-accent-dim text-accent'
|
||||||
|
: 'text-muted hover:text-main hover:bg-panel-elevated'
|
||||||
|
}`}
|
||||||
|
>
|
||||||
|
{isTranscribing ? (
|
||||||
|
<Loader2 className="w-3 h-3 animate-spin" />
|
||||||
|
) : isListening ? (
|
||||||
|
<MicOff className="w-3 h-3" />
|
||||||
|
) : (
|
||||||
|
<Mic className="w-3 h-3" />
|
||||||
|
)}
|
||||||
|
</button>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
|
||||||
{/* Instant Send Button */}
|
{/* Instant Send Button */}
|
||||||
<button
|
<button
|
||||||
|
|||||||
@@ -0,0 +1,133 @@
|
|||||||
|
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||||
|
import { act, renderHook, waitFor } from '@testing-library/react';
|
||||||
|
|
||||||
|
const getAuthStatus = vi.fn();
|
||||||
|
const transcribe = vi.fn();
|
||||||
|
vi.mock('../api', () => ({ api: { getAuthStatus: (...a: any[]) => getAuthStatus(...a), transcribe: (...a: any[]) => transcribe(...a) } }));
|
||||||
|
|
||||||
|
import { useVoiceInput } from '../useVoiceInput';
|
||||||
|
import { resetVoiceConfig } from '../voice';
|
||||||
|
|
||||||
|
class FakeMediaRecorder {
|
||||||
|
static isTypeSupported = (t: string) => t === 'audio/webm;codecs=opus';
|
||||||
|
state: 'inactive' | 'recording' = 'inactive';
|
||||||
|
mimeType = 'audio/webm;codecs=opus';
|
||||||
|
ondataavailable: ((e: { data: Blob }) => void) | null = null;
|
||||||
|
onstop: (() => void) | null = null;
|
||||||
|
constructor(public stream: any) {}
|
||||||
|
start() {
|
||||||
|
this.state = 'recording';
|
||||||
|
}
|
||||||
|
stop() {
|
||||||
|
this.state = 'inactive';
|
||||||
|
this.ondataavailable?.({ data: new Blob(['audio'], { type: this.mimeType }) });
|
||||||
|
this.onstop?.();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const stopTrack = vi.fn();
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
resetVoiceConfig();
|
||||||
|
getAuthStatus.mockReset();
|
||||||
|
transcribe.mockReset();
|
||||||
|
stopTrack.mockReset();
|
||||||
|
(window as any).MediaRecorder = FakeMediaRecorder;
|
||||||
|
Object.defineProperty(navigator, 'mediaDevices', {
|
||||||
|
configurable: true,
|
||||||
|
value: { getUserMedia: vi.fn(async () => ({ getTracks: () => [{ stop: stopTrack }] })) },
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
delete (window as any).MediaRecorder;
|
||||||
|
delete (window as any).SpeechRecognition;
|
||||||
|
vi.useRealTimers();
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('useVoiceInput', () => {
|
||||||
|
it('server mode: records, sends the audio to the own server and returns its text', async () => {
|
||||||
|
getAuthStatus.mockResolvedValue({ voice: { mode: 'server', maxSeconds: 600 } });
|
||||||
|
transcribe.mockResolvedValue('Milch kaufen');
|
||||||
|
const onText = vi.fn();
|
||||||
|
const { result } = renderHook(() => useVoiceInput({ onText }));
|
||||||
|
await waitFor(() => expect(result.current.mode).toBe('server'));
|
||||||
|
expect(result.current.available).toBe(true);
|
||||||
|
|
||||||
|
await act(async () => {
|
||||||
|
await result.current.start();
|
||||||
|
});
|
||||||
|
expect(result.current.state).toBe('listening');
|
||||||
|
|
||||||
|
await act(async () => {
|
||||||
|
await result.current.stop();
|
||||||
|
});
|
||||||
|
expect(transcribe).toHaveBeenCalledTimes(1);
|
||||||
|
expect((transcribe.mock.calls[0][0] as Blob).type).toBe('audio/webm;codecs=opus');
|
||||||
|
expect(onText).toHaveBeenCalledWith('Milch kaufen', true);
|
||||||
|
expect(result.current.state).toBe('idle');
|
||||||
|
expect(stopTrack).toHaveBeenCalled(); // microphone released
|
||||||
|
});
|
||||||
|
|
||||||
|
it('server mode: cancel throws the recording away', async () => {
|
||||||
|
getAuthStatus.mockResolvedValue({ voice: { mode: 'server', maxSeconds: 600 } });
|
||||||
|
const onText = vi.fn();
|
||||||
|
const { result } = renderHook(() => useVoiceInput({ onText }));
|
||||||
|
await waitFor(() => expect(result.current.mode).toBe('server'));
|
||||||
|
await act(async () => {
|
||||||
|
await result.current.start();
|
||||||
|
});
|
||||||
|
act(() => result.current.cancel());
|
||||||
|
expect(transcribe).not.toHaveBeenCalled();
|
||||||
|
expect(onText).not.toHaveBeenCalled();
|
||||||
|
expect(result.current.state).toBe('idle');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('server mode: a failing server reports an error instead of text', async () => {
|
||||||
|
getAuthStatus.mockResolvedValue({ voice: { mode: 'server', maxSeconds: 600 } });
|
||||||
|
transcribe.mockRejectedValue(new Error('Spracherkennung fehlgeschlagen'));
|
||||||
|
const onText = vi.fn();
|
||||||
|
const onError = vi.fn();
|
||||||
|
const { result } = renderHook(() => useVoiceInput({ onText, onError }));
|
||||||
|
await waitFor(() => expect(result.current.mode).toBe('server'));
|
||||||
|
await act(async () => {
|
||||||
|
await result.current.start();
|
||||||
|
});
|
||||||
|
await act(async () => {
|
||||||
|
await result.current.stop();
|
||||||
|
});
|
||||||
|
expect(onError).toHaveBeenCalledWith('Spracherkennung fehlgeschlagen');
|
||||||
|
expect(onText).not.toHaveBeenCalled();
|
||||||
|
expect(result.current.state).toBe('idle');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('server mode: stops by itself after the maximum length', async () => {
|
||||||
|
getAuthStatus.mockResolvedValue({ voice: { mode: 'server', maxSeconds: 5 } });
|
||||||
|
transcribe.mockResolvedValue('Zu lang');
|
||||||
|
const onText = vi.fn();
|
||||||
|
const { result } = renderHook(() => useVoiceInput({ onText }));
|
||||||
|
await waitFor(() => expect(result.current.mode).toBe('server'));
|
||||||
|
vi.useFakeTimers();
|
||||||
|
await act(async () => {
|
||||||
|
await result.current.start();
|
||||||
|
});
|
||||||
|
await act(async () => {
|
||||||
|
vi.advanceTimersByTime(5000);
|
||||||
|
});
|
||||||
|
vi.useRealTimers();
|
||||||
|
await waitFor(() => expect(onText).toHaveBeenCalledWith('Zu lang', true));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('hides the microphone when switched off, and without browser recognition', async () => {
|
||||||
|
getAuthStatus.mockResolvedValue({ voice: { mode: 'off', maxSeconds: 600 } });
|
||||||
|
const off = renderHook(() => useVoiceInput({ onText: vi.fn() }));
|
||||||
|
await waitFor(() => expect(off.result.current.mode).toBe('off'));
|
||||||
|
expect(off.result.current.available).toBe(false);
|
||||||
|
|
||||||
|
resetVoiceConfig();
|
||||||
|
getAuthStatus.mockResolvedValue({ voice: { mode: 'browser', maxSeconds: 600 } });
|
||||||
|
const none = renderHook(() => useVoiceInput({ onText: vi.fn() }));
|
||||||
|
await waitFor(() => expect(none.result.current.mode).toBe('unsupported'));
|
||||||
|
expect(none.result.current.available).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,200 @@
|
|||||||
|
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||||
|
import { api } from './api';
|
||||||
|
import { browserRecognition, currentVoiceConfig, EffectiveVoiceMode, loadVoiceConfig, VoiceRecorder } from './voice';
|
||||||
|
|
||||||
|
export type VoiceState = 'idle' | 'listening' | 'transcribing';
|
||||||
|
|
||||||
|
export interface VoiceInputOptions {
|
||||||
|
/** Text of the current recording; in browser mode repeatedly while speaking (final=false), always once at the end (final=true). */
|
||||||
|
onText: (text: string, final: boolean) => void;
|
||||||
|
onError?: (message: string) => void;
|
||||||
|
/** Browser mode: keep listening through pauses (hold-to-record) instead of stopping after a sentence. */
|
||||||
|
continuous?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Microphone input for one button: start, stop (text arrives), cancel (nothing arrives). How the text
|
||||||
|
* is made depends on the server setting, see lib/voice.ts. mode is null until known, "off" or
|
||||||
|
* "unsupported" means: do not show a microphone.
|
||||||
|
*/
|
||||||
|
export function useVoiceInput({ onText, onError, continuous = false }: VoiceInputOptions) {
|
||||||
|
const [mode, setMode] = useState<EffectiveVoiceMode | null>(null);
|
||||||
|
const [state, setState] = useState<VoiceState>('idle');
|
||||||
|
const maxSecondsRef = useRef(600);
|
||||||
|
const recorderRef = useRef<VoiceRecorder | null>(null);
|
||||||
|
const recognitionRef = useRef<any>(null);
|
||||||
|
const timerRef = useRef<number | null>(null);
|
||||||
|
const cancelledRef = useRef(false);
|
||||||
|
const transcriptRef = useRef('');
|
||||||
|
const mountedRef = useRef(true);
|
||||||
|
const onTextRef = useRef(onText);
|
||||||
|
onTextRef.current = onText;
|
||||||
|
const onErrorRef = useRef(onError);
|
||||||
|
onErrorRef.current = onError;
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
mountedRef.current = true;
|
||||||
|
void loadVoiceConfig().then((cfg) => {
|
||||||
|
if (!mountedRef.current) return;
|
||||||
|
setMode(cfg.mode);
|
||||||
|
maxSecondsRef.current = cfg.maxSeconds;
|
||||||
|
});
|
||||||
|
return () => {
|
||||||
|
mountedRef.current = false;
|
||||||
|
if (timerRef.current !== null) window.clearTimeout(timerRef.current);
|
||||||
|
recorderRef.current?.cancel();
|
||||||
|
try {
|
||||||
|
recognitionRef.current?.abort?.();
|
||||||
|
} catch {
|
||||||
|
// ignore
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const setStateSafe = (s: VoiceState) => {
|
||||||
|
if (mountedRef.current) setState(s);
|
||||||
|
};
|
||||||
|
const fail = (message: string) => {
|
||||||
|
setStateSafe('idle');
|
||||||
|
onErrorRef.current?.(message);
|
||||||
|
};
|
||||||
|
const clearTimer = () => {
|
||||||
|
if (timerRef.current !== null) window.clearTimeout(timerRef.current);
|
||||||
|
timerRef.current = null;
|
||||||
|
};
|
||||||
|
|
||||||
|
const stop = useCallback(async () => {
|
||||||
|
clearTimer();
|
||||||
|
if (recognitionRef.current) {
|
||||||
|
try {
|
||||||
|
recognitionRef.current.stop();
|
||||||
|
} catch {
|
||||||
|
// onend still follows
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const rec = recorderRef.current;
|
||||||
|
if (!rec) return;
|
||||||
|
recorderRef.current = null;
|
||||||
|
const blob = await rec.stop();
|
||||||
|
if (cancelledRef.current) return;
|
||||||
|
if (!blob) {
|
||||||
|
setStateSafe('idle');
|
||||||
|
onTextRef.current('', true);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
setStateSafe('transcribing');
|
||||||
|
try {
|
||||||
|
const text = await api.transcribe(blob);
|
||||||
|
setStateSafe('idle');
|
||||||
|
onTextRef.current(text, true);
|
||||||
|
} catch (err: any) {
|
||||||
|
fail(err?.message || 'Spracherkennung fehlgeschlagen');
|
||||||
|
}
|
||||||
|
// fail only touches refs and state setters
|
||||||
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const startServer = async () => {
|
||||||
|
const rec = new VoiceRecorder();
|
||||||
|
recorderRef.current = rec;
|
||||||
|
setStateSafe('listening');
|
||||||
|
try {
|
||||||
|
await rec.start();
|
||||||
|
} catch {
|
||||||
|
recorderRef.current = null;
|
||||||
|
fail('Kein Zugriff auf das Mikrofon (Berechtigung, oder die Seite läuft nicht über HTTPS)');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (cancelledRef.current || recorderRef.current !== rec) {
|
||||||
|
rec.cancel();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
timerRef.current = window.setTimeout(() => void stop(), maxSecondsRef.current * 1000);
|
||||||
|
};
|
||||||
|
|
||||||
|
const startBrowser = () => {
|
||||||
|
const Recognition = browserRecognition();
|
||||||
|
if (!Recognition) {
|
||||||
|
fail('Spracheingabe wird von diesem Browser nicht unterstützt');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const recognition = new Recognition();
|
||||||
|
recognition.lang = 'de-DE';
|
||||||
|
recognition.interimResults = true;
|
||||||
|
recognition.continuous = continuous;
|
||||||
|
recognition.onresult = (event: any) => {
|
||||||
|
let combined = '';
|
||||||
|
for (let i = 0; i < event.results.length; i++) combined += event.results[i][0].transcript;
|
||||||
|
transcriptRef.current = combined;
|
||||||
|
onTextRef.current(combined, false);
|
||||||
|
};
|
||||||
|
recognition.onerror = (event: any) => {
|
||||||
|
if (event?.error === 'not-allowed' || event?.error === 'service-not-allowed') {
|
||||||
|
cancelledRef.current = true;
|
||||||
|
recognitionRef.current = null;
|
||||||
|
fail('Mikrofon nicht freigegeben');
|
||||||
|
}
|
||||||
|
};
|
||||||
|
recognition.onend = () => {
|
||||||
|
clearTimer();
|
||||||
|
recognitionRef.current = null;
|
||||||
|
if (cancelledRef.current) return;
|
||||||
|
setStateSafe('idle');
|
||||||
|
onTextRef.current(transcriptRef.current, true);
|
||||||
|
};
|
||||||
|
recognitionRef.current = recognition;
|
||||||
|
setStateSafe('listening');
|
||||||
|
try {
|
||||||
|
recognition.start();
|
||||||
|
timerRef.current = window.setTimeout(() => void stop(), maxSecondsRef.current * 1000);
|
||||||
|
} catch {
|
||||||
|
recognitionRef.current = null;
|
||||||
|
fail('Spracheingabe konnte nicht gestartet werden');
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const start = useCallback(async () => {
|
||||||
|
if (recorderRef.current || recognitionRef.current) return;
|
||||||
|
cancelledRef.current = false;
|
||||||
|
transcriptRef.current = '';
|
||||||
|
const current = mode ?? currentVoiceConfig().mode;
|
||||||
|
if (current === 'server') await startServer();
|
||||||
|
else if (current === 'browser') startBrowser();
|
||||||
|
else fail(current === 'off' ? 'Spracheingabe ist ausgeschaltet' : 'Spracheingabe wird von diesem Browser nicht unterstützt');
|
||||||
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||||
|
}, [mode, continuous]);
|
||||||
|
|
||||||
|
const cancel = useCallback(() => {
|
||||||
|
cancelledRef.current = true;
|
||||||
|
clearTimer();
|
||||||
|
recorderRef.current?.cancel();
|
||||||
|
recorderRef.current = null;
|
||||||
|
try {
|
||||||
|
recognitionRef.current?.abort?.();
|
||||||
|
} catch {
|
||||||
|
// ignore
|
||||||
|
}
|
||||||
|
recognitionRef.current = null;
|
||||||
|
setStateSafe('idle');
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const toggle = useCallback(() => {
|
||||||
|
if (state === 'listening') void stop();
|
||||||
|
else if (state === 'idle') void start();
|
||||||
|
}, [state, start, stop]);
|
||||||
|
|
||||||
|
/** Current input level 0..1 for waves (server mode), null when unknown. */
|
||||||
|
const level = useCallback(() => recorderRef.current?.level() ?? null, []);
|
||||||
|
|
||||||
|
return {
|
||||||
|
mode,
|
||||||
|
available: mode === 'server' || mode === 'browser',
|
||||||
|
state,
|
||||||
|
start,
|
||||||
|
stop,
|
||||||
|
cancel,
|
||||||
|
toggle,
|
||||||
|
level,
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,157 @@
|
|||||||
|
// Voice input for all places with a microphone button. The server decides how it works (STT_* in the
|
||||||
|
// environment, reported by /auth/status):
|
||||||
|
// server record with MediaRecorder, the own Whisper server turns it into text after stopping
|
||||||
|
// browser the browser's speech recognition (Chrome, Edge, Safari; text appears while speaking)
|
||||||
|
// off no voice input
|
||||||
|
import { api } from './api';
|
||||||
|
import type { VoiceMode } from './types';
|
||||||
|
|
||||||
|
export type EffectiveVoiceMode = VoiceMode | 'unsupported';
|
||||||
|
|
||||||
|
export interface VoiceConfig {
|
||||||
|
mode: EffectiveVoiceMode;
|
||||||
|
maxSeconds: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
let configPromise: Promise<VoiceConfig> | null = null;
|
||||||
|
let loadedConfig: VoiceConfig | null = null;
|
||||||
|
|
||||||
|
export function browserRecognition(): any {
|
||||||
|
if (typeof window === 'undefined') return null;
|
||||||
|
return (window as any).SpeechRecognition || (window as any).webkitSpeechRecognition || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function canRecord(): boolean {
|
||||||
|
return typeof window !== 'undefined' && typeof (window as any).MediaRecorder !== 'undefined' && !!navigator.mediaDevices?.getUserMedia;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** What this browser can actually do with the server's setting. */
|
||||||
|
export function effectiveMode(mode: VoiceMode | undefined): EffectiveVoiceMode {
|
||||||
|
if (mode === 'off') return 'off';
|
||||||
|
if (mode === 'server') return canRecord() ? 'server' : 'unsupported';
|
||||||
|
return browserRecognition() ? 'browser' : 'unsupported';
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Loaded once per page; the setting only changes with a server restart. */
|
||||||
|
export function loadVoiceConfig(): Promise<VoiceConfig> {
|
||||||
|
if (!configPromise) {
|
||||||
|
configPromise = api
|
||||||
|
.getAuthStatus()
|
||||||
|
.then((st) => {
|
||||||
|
loadedConfig = { mode: effectiveMode(st.voice?.mode), maxSeconds: st.voice?.maxSeconds || 600 };
|
||||||
|
return loadedConfig;
|
||||||
|
})
|
||||||
|
.catch(() => {
|
||||||
|
configPromise = null; // offline at start: ask again next time
|
||||||
|
return { mode: effectiveMode('browser'), maxSeconds: 600 };
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return configPromise;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The setting if already loaded; until then the browser's recognition (a button must react at once). */
|
||||||
|
export function currentVoiceConfig(): VoiceConfig {
|
||||||
|
return loadedConfig ?? { mode: effectiveMode('browser'), maxSeconds: 600 };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** For tests. */
|
||||||
|
export function resetVoiceConfig(): void {
|
||||||
|
configPromise = null;
|
||||||
|
loadedConfig = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Containers in order of preference; Safari only records MP4/AAC
|
||||||
|
const RECORDING_TYPES = ['audio/webm;codecs=opus', 'audio/webm', 'audio/mp4', 'audio/ogg;codecs=opus'];
|
||||||
|
|
||||||
|
function recordingType(): string | undefined {
|
||||||
|
const MR = (window as any).MediaRecorder;
|
||||||
|
if (!MR?.isTypeSupported) return undefined;
|
||||||
|
return RECORDING_TYPES.find((t) => MR.isTypeSupported(t));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One recording from the microphone, with a live input level for wave animations. stop() resolves
|
||||||
|
* with the audio; cancel() throws it away.
|
||||||
|
*/
|
||||||
|
export class VoiceRecorder {
|
||||||
|
private stream: MediaStream | null = null;
|
||||||
|
private recorder: MediaRecorder | null = null;
|
||||||
|
private chunks: Blob[] = [];
|
||||||
|
private audioCtx: AudioContext | null = null;
|
||||||
|
private analyser: AnalyserNode | null = null;
|
||||||
|
private samples: Uint8Array | null = null;
|
||||||
|
private done: ((blob: Blob | null) => void) | null = null;
|
||||||
|
|
||||||
|
async start(): Promise<void> {
|
||||||
|
this.stream = await navigator.mediaDevices.getUserMedia({
|
||||||
|
audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true },
|
||||||
|
});
|
||||||
|
const type = recordingType();
|
||||||
|
this.recorder = new MediaRecorder(this.stream, type ? { mimeType: type } : undefined);
|
||||||
|
this.chunks = [];
|
||||||
|
this.recorder.ondataavailable = (e) => {
|
||||||
|
if (e.data.size > 0) this.chunks.push(e.data);
|
||||||
|
};
|
||||||
|
this.recorder.onstop = () => {
|
||||||
|
const blob = this.chunks.length ? new Blob(this.chunks, { type: this.recorder?.mimeType || type || 'audio/webm' }) : null;
|
||||||
|
this.release();
|
||||||
|
this.done?.(blob);
|
||||||
|
this.done = null;
|
||||||
|
};
|
||||||
|
this.recorder.start(1000);
|
||||||
|
try {
|
||||||
|
const Ctx = (window as any).AudioContext || (window as any).webkitAudioContext;
|
||||||
|
this.audioCtx = new Ctx();
|
||||||
|
const source = this.audioCtx!.createMediaStreamSource(this.stream);
|
||||||
|
this.analyser = this.audioCtx!.createAnalyser();
|
||||||
|
this.analyser.fftSize = 512;
|
||||||
|
source.connect(this.analyser);
|
||||||
|
this.samples = new Uint8Array(this.analyser.fftSize);
|
||||||
|
void this.audioCtx!.resume?.();
|
||||||
|
} catch {
|
||||||
|
this.analyser = null; // no level, the recording still works
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Input level 0..1 (roughly loudness), or null when unknown. */
|
||||||
|
level(): number | null {
|
||||||
|
if (!this.analyser || !this.samples) return null;
|
||||||
|
this.analyser.getByteTimeDomainData(this.samples as Uint8Array<ArrayBuffer>);
|
||||||
|
let sum = 0;
|
||||||
|
for (const v of this.samples) {
|
||||||
|
const x = (v - 128) / 128;
|
||||||
|
sum += x * x;
|
||||||
|
}
|
||||||
|
return Math.min(1, Math.sqrt(sum / this.samples.length) * 4);
|
||||||
|
}
|
||||||
|
|
||||||
|
stop(): Promise<Blob | null> {
|
||||||
|
return new Promise((resolve) => {
|
||||||
|
if (!this.recorder || this.recorder.state === 'inactive') {
|
||||||
|
this.release();
|
||||||
|
resolve(null);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.done = resolve;
|
||||||
|
this.recorder.stop();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
cancel(): void {
|
||||||
|
this.done = null;
|
||||||
|
if (this.recorder && this.recorder.state !== 'inactive') {
|
||||||
|
this.recorder.onstop = () => this.release();
|
||||||
|
this.recorder.stop();
|
||||||
|
} else {
|
||||||
|
this.release();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private release(): void {
|
||||||
|
this.stream?.getTracks().forEach((t) => t.stop());
|
||||||
|
this.stream = null;
|
||||||
|
void this.audioCtx?.close?.().catch(() => {});
|
||||||
|
this.audioCtx = null;
|
||||||
|
this.analyser = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in new issue
Block a user