Fix Android voice bug - STT restart after TTS
PROBLEM: startListening() triggers spurious AppState change to "background" on Android, causing voice session to stop immediately after Julia responds. ROOT CAUSE: React Native AppState bug - requesting audio focus triggers false background event. SOLUTION: - Added sttStartingIgnoreAppStateRef flag - Ignore AppState changes for 200ms after startListening() call - Protects against false session termination during STT initialization CHANGES: - app/(tabs)/_layout.tsx: Added Android workaround with 200ms protection window - VOICE_DEBUG_GUIDE.md: Documented bug, workaround, and expected logs RESULT: Voice session now continues correctly after Julia's response on Android. STT successfully restarts and user can speak again without manual restart.
This commit is contained in:
parent
3ec0f5dae2
commit
81a0c59060
321
VOICE_DEBUG_GUIDE.md
Normal file
321
VOICE_DEBUG_GUIDE.md
Normal file
@ -0,0 +1,321 @@
|
|||||||
|
# Voice Recognition Debug Guide
|
||||||
|
|
||||||
|
## Как работает голосовая система на Android vs iOS
|
||||||
|
|
||||||
|
### Основная проблема: Audio Focus на Android
|
||||||
|
|
||||||
|
**Android:** STT (Speech-to-Text) и TTS (Text-to-Speech) **НЕ МОГУТ** работать одновременно из-за конфликта Audio Focus. Когда TTS начинает говорить, она захватывает аудио-фокус, и STT автоматически останавливается системой.
|
||||||
|
|
||||||
|
**iOS:** STT и TTS **МОГУТ** работать одновременно. STT может слушать даже во время воспроизведения TTS (что позволяет прерывать Julia голосом).
|
||||||
|
|
||||||
|
### КРИТИЧЕСКАЯ проблема Android: startListening() → AppState background
|
||||||
|
|
||||||
|
**На Android обнаружен баг React Native:**
|
||||||
|
- При вызове `startListening()` AppState кратковременно переходит в `"background"` или `"inactive"`
|
||||||
|
- Это происходит из-за запроса аудио-фокуса и системных разрешений
|
||||||
|
- **НЕ связано** с действиями пользователя (блокировка экрана, переключение приложений)
|
||||||
|
|
||||||
|
**Workaround (реализован в коде):**
|
||||||
|
- Добавлен флаг `sttStartingIgnoreAppStateRef`
|
||||||
|
- Игнорируем AppState changes на 200ms после вызова `startListening()`
|
||||||
|
- Защищает от ложной остановки сессии во время запуска STT
|
||||||
|
|
||||||
|
**Логи при срабатывании workaround:**
|
||||||
|
```
|
||||||
|
LOG [Android] [TabLayout] 🛡️ Ignoring AppState changes for 200ms (STT start workaround)
|
||||||
|
LOG [Android] [TabLayout] ▶️ STARTING STT... (status: listening)
|
||||||
|
LOG [Android] [TabLayout] 📱 AppState changed to: "background"
|
||||||
|
LOG [Android] [TabLayout] 🛡️ IGNORING AppState change (STT start protection active)
|
||||||
|
LOG [Android] [TabLayout] ✅ STT started successfully
|
||||||
|
LOG [Android] [TabLayout] ✅ AppState monitoring resumed
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Нормальный Flow (Должен работать так)
|
||||||
|
|
||||||
|
### 1️⃣ **Юзер нажимает кнопку "Start"**
|
||||||
|
|
||||||
|
**Android:**
|
||||||
|
```
|
||||||
|
[Android] [TabLayout] 🎯 FAB PRESSED - isSessionActive: false
|
||||||
|
[Android] [TabLayout] ▶️ STARTING session (FAB start)
|
||||||
|
[Android] [VoiceContext] 🎤 STARTING voice session
|
||||||
|
[Android] [VoiceContext] → Session initialized, status=listening
|
||||||
|
[Android] [TabLayout] 🎤 Voice session STARTED - starting STT
|
||||||
|
[Android] [TabLayout] ▶️ STARTING STT... (status: listening)
|
||||||
|
[Android] [TabLayout] ✅ STT started successfully
|
||||||
|
```
|
||||||
|
|
||||||
|
**iOS:**
|
||||||
|
```
|
||||||
|
[iOS] [TabLayout] 🎯 FAB PRESSED - isSessionActive: false
|
||||||
|
[iOS] [TabLayout] ▶️ STARTING session (FAB start)
|
||||||
|
[iOS] [VoiceContext] 🎤 STARTING voice session
|
||||||
|
[iOS] [VoiceContext] → Session initialized, status=listening
|
||||||
|
[iOS] [TabLayout] 🎤 Voice session STARTED - starting STT
|
||||||
|
[iOS] [TabLayout] ▶️ STARTING STT... (status: listening)
|
||||||
|
[iOS] [TabLayout] ✅ STT started successfully
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2️⃣ **Юзер говорит что-то**
|
||||||
|
|
||||||
|
**Android:**
|
||||||
|
```
|
||||||
|
[Android] [TabLayout] handleSpeechResult - isFinal: false, status: listening, transcript: "hello..."
|
||||||
|
[Android] [TabLayout] → Updating PARTIAL transcript
|
||||||
|
[Android] [TabLayout] Partial changed: "" → "hello"
|
||||||
|
[Android] [TabLayout] → Started 2s silence timer
|
||||||
|
```
|
||||||
|
|
||||||
|
**iOS:** (то же самое)
|
||||||
|
|
||||||
|
### 3️⃣ **2 секунды тишины → auto-stop**
|
||||||
|
|
||||||
|
**Android:**
|
||||||
|
```
|
||||||
|
[Android] [TabLayout] 🤖 AUTO-STOP: 2s silence - stopping STT
|
||||||
|
[Android] [TabLayout] handleSTTEnd - sessionActive: true, status: listening
|
||||||
|
[Android] [TabLayout] → shouldRestartSTT set to TRUE
|
||||||
|
[Android] [TabLayout] handleSpeechResult - isFinal: true, status: listening, transcript: "hello"
|
||||||
|
[Android] [TabLayout] → Processing FINAL transcript, sending to API
|
||||||
|
```
|
||||||
|
|
||||||
|
**iOS:** (то же самое, но с префиксом `[iOS]`)
|
||||||
|
|
||||||
|
### 4️⃣ **API запрос**
|
||||||
|
|
||||||
|
**Android:**
|
||||||
|
```
|
||||||
|
[Android] [VoiceContext] 📤 Sending transcript to API (ask_wellnuo_ai): "hello"
|
||||||
|
[Android] [VoiceContext] 🔑 Getting API token...
|
||||||
|
[Android] [VoiceContext] ✅ Token obtained
|
||||||
|
[Android] [VoiceContext] 📝 Normalized question: "how is dad doing"
|
||||||
|
[Android] [VoiceContext] 📡 Using API type: ask_wellnuo_ai, deployment: 21
|
||||||
|
[Android] [VoiceContext] 🌐 Sending API request...
|
||||||
|
[Android] [TabLayout] Status transition: listening → processing, sttIsListening: true
|
||||||
|
[Android] [TabLayout] ⏸️ Stopping STT during processing (echo prevention)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Важно на Android:** STT останавливается во время `processing`, чтобы не слышать эхо от TTS.
|
||||||
|
|
||||||
|
### 5️⃣ **API ответ получен**
|
||||||
|
|
||||||
|
**Android:**
|
||||||
|
```
|
||||||
|
[Android] [VoiceContext] 📥 API response received, parsing...
|
||||||
|
[Android] [VoiceContext] ✅ API SUCCESS: "Your dad Ferdinand is doing well. He spent most of his time..."
|
||||||
|
[Android] [VoiceContext] 🔊 Starting TTS for response...
|
||||||
|
[Android] [VoiceContext] 🔊 Starting TTS: "Your dad Ferdinand is doing well..."
|
||||||
|
[Android] [VoiceContext] ▶️ TTS playback STARTED
|
||||||
|
[Android] [TabLayout] Status transition: processing → speaking, sttIsListening: false
|
||||||
|
```
|
||||||
|
|
||||||
|
**iOS:** (то же самое)
|
||||||
|
|
||||||
|
### 6️⃣ **TTS заканчивает говорить**
|
||||||
|
|
||||||
|
**Android:**
|
||||||
|
```
|
||||||
|
[Android] [VoiceContext] ✅ TTS playback COMPLETED
|
||||||
|
[Android] [VoiceContext] → isSpeaking = false (immediate - audio focus release)
|
||||||
|
[Android] [VoiceContext] → status = listening (ready for next input)
|
||||||
|
[Android] [VoiceContext] ✅ TTS completed
|
||||||
|
[Android] [TabLayout] Status transition: speaking → listening, sttIsListening: false
|
||||||
|
[Android] [TabLayout] 🔄 TTS FINISHED - preparing to restart STT
|
||||||
|
[Android] [TabLayout] ⏱️ Waiting 50ms before restarting STT (audio focus release)
|
||||||
|
```
|
||||||
|
|
||||||
|
**iOS:**
|
||||||
|
```
|
||||||
|
[iOS] [VoiceContext] ✅ TTS playback COMPLETED
|
||||||
|
[iOS] [VoiceContext] ⏱️ Delaying isSpeaking=false by 300ms (match STT restart)
|
||||||
|
[iOS] [VoiceContext] → status = listening (ready for next input)
|
||||||
|
[iOS] [VoiceContext] ✅ TTS completed
|
||||||
|
[iOS] [TabLayout] Status transition: speaking → listening, sttIsListening: false
|
||||||
|
[iOS] [TabLayout] 🔄 TTS FINISHED - preparing to restart STT
|
||||||
|
[iOS] [TabLayout] ⏱️ Waiting 300ms before restarting STT (audio focus release)
|
||||||
|
[iOS] [VoiceContext] → isSpeaking = false (after 300ms delay)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Ключевая разница:**
|
||||||
|
- **Android:** `isSpeaking = false` **сразу** (50ms delay для STT restart)
|
||||||
|
- **iOS:** `isSpeaking = false` через **300ms** (плавный переход, matches STT restart)
|
||||||
|
|
||||||
|
### 7️⃣ **STT рестартует после TTS**
|
||||||
|
|
||||||
|
**Android:**
|
||||||
|
```
|
||||||
|
[Android] [TabLayout] ⏰ Delay complete - restarting STT now
|
||||||
|
[Android] [TabLayout] ▶️ STARTING STT... (status: listening)
|
||||||
|
[Android] [TabLayout] ✅ STT started successfully
|
||||||
|
```
|
||||||
|
|
||||||
|
**iOS:** (то же самое)
|
||||||
|
|
||||||
|
### 8️⃣ **Юзер может говорить снова → повтор с шага 2**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Что НЕ ДОЛЖНО происходить (баги)
|
||||||
|
|
||||||
|
### ❌ БАГ 1: STT не рестартует после TTS (Android)
|
||||||
|
|
||||||
|
**Симптомы:**
|
||||||
|
```
|
||||||
|
[Android] [VoiceContext] ✅ TTS playback COMPLETED
|
||||||
|
[Android] [TabLayout] Status transition: speaking → listening, sttIsListening: false
|
||||||
|
[Android] [TabLayout] 🔄 TTS FINISHED - preparing to restart STT
|
||||||
|
[Android] [TabLayout] ⏱️ Waiting 50ms before restarting STT (audio focus release)
|
||||||
|
[Android] [TabLayout] ⏰ Delay complete - restarting STT now
|
||||||
|
[Android] [TabLayout] safeStartSTT - already listening or starting, skipping ❌ ПРОБЛЕМА!
|
||||||
|
```
|
||||||
|
|
||||||
|
**Причина:** `sttIsListening` или `sttStartingRef.current` остались `true`, хотя STT на самом деле не работает.
|
||||||
|
|
||||||
|
**Решение:** Убедиться что `stopListening()` вызывается корректно и флаги сбрасываются.
|
||||||
|
|
||||||
|
### ❌ БАГ 2: STT не останавливается во время TTS (эхо)
|
||||||
|
|
||||||
|
**Симптомы:**
|
||||||
|
```
|
||||||
|
[Android] [TabLayout] Status transition: processing → speaking, sttIsListening: true ❌ ПРОБЛЕМА!
|
||||||
|
```
|
||||||
|
|
||||||
|
**Причина:** STT не остановилось при переходе в `speaking`, и будет слышать Julia как эхо.
|
||||||
|
|
||||||
|
**Решение:** Проверить что `stopListening()` вызывается при `status === 'speaking'`.
|
||||||
|
|
||||||
|
### ❌ БАГ 3: isSpeaking не сбрасывается
|
||||||
|
|
||||||
|
**Симптомы (Android):**
|
||||||
|
```
|
||||||
|
[Android] [VoiceContext] ✅ TTS playback COMPLETED
|
||||||
|
[Android] [VoiceContext] → isSpeaking = false (immediate - audio focus release)
|
||||||
|
... но FAB остаётся зелёным (isSpeaking === true в UI)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Причина:** `setIsSpeaking(false)` не отрабатывает или есть race condition.
|
||||||
|
|
||||||
|
**Решение:** Проверить что `setIsSpeaking(false)` вызывается в `onDone` TTS callback.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Как читать логи
|
||||||
|
|
||||||
|
### Префиксы платформ:
|
||||||
|
- `[iOS]` — событие происходит на iOS
|
||||||
|
- `[Android]` — событие происходит на Android
|
||||||
|
|
||||||
|
### Префиксы модулей:
|
||||||
|
- `[TabLayout]` — события из `app/(tabs)/_layout.tsx` (STT orchestration)
|
||||||
|
- `[VoiceContext]` — события из `contexts/VoiceContext.tsx` (API + TTS)
|
||||||
|
|
||||||
|
### Эмодзи:
|
||||||
|
- 🎯 — FAB нажат
|
||||||
|
- 🎤 — Voice session start/stop
|
||||||
|
- ▶️ — STT start
|
||||||
|
- ⏸️ — STT stop
|
||||||
|
- 📤 — Sending to API
|
||||||
|
- 📥 — API response
|
||||||
|
- 🔊 — TTS start/playback
|
||||||
|
- ✅ — Success
|
||||||
|
- ❌ — Error
|
||||||
|
- ⚠️ — Warning
|
||||||
|
- 🔄 — Restart/retry
|
||||||
|
- ⏱️ — Timer/delay
|
||||||
|
- ⏰ — Timer triggered
|
||||||
|
- 🛑 — Force stop
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Как тестировать
|
||||||
|
|
||||||
|
### Тест 1: Базовый цикл (1 вопрос → 1 ответ → 1 вопрос)
|
||||||
|
|
||||||
|
1. Нажать FAB (зелёная кнопка)
|
||||||
|
2. Сказать: "Hello"
|
||||||
|
3. Подождать 2 секунды (auto-stop)
|
||||||
|
4. **Проверить:** Julia отвечает
|
||||||
|
5. **Проверить:** После ответа Julia, STT рестартует автоматически
|
||||||
|
6. Сказать: "How is dad?"
|
||||||
|
7. **Проверить:** Julia отвечает снова
|
||||||
|
|
||||||
|
**Ожидаемые логи (Android):**
|
||||||
|
```
|
||||||
|
[Android] [TabLayout] 🎯 FAB PRESSED - isSessionActive: false
|
||||||
|
[Android] [TabLayout] ▶️ STARTING session (FAB start)
|
||||||
|
[Android] [TabLayout] ▶️ STARTING STT... (status: listening)
|
||||||
|
[Android] [TabLayout] ✅ STT started successfully
|
||||||
|
... пользователь говорит ...
|
||||||
|
[Android] [TabLayout] 🤖 AUTO-STOP: 2s silence - stopping STT
|
||||||
|
[Android] [TabLayout] → Processing FINAL transcript, sending to API
|
||||||
|
[Android] [VoiceContext] 📤 Sending transcript to API
|
||||||
|
[Android] [TabLayout] ⏸️ Stopping STT during processing (echo prevention)
|
||||||
|
[Android] [VoiceContext] ✅ API SUCCESS
|
||||||
|
[Android] [VoiceContext] 🔊 Starting TTS for response...
|
||||||
|
[Android] [VoiceContext] ▶️ TTS playback STARTED
|
||||||
|
... Julia говорит ...
|
||||||
|
[Android] [VoiceContext] ✅ TTS playback COMPLETED
|
||||||
|
[Android] [TabLayout] 🔄 TTS FINISHED - preparing to restart STT
|
||||||
|
[Android] [TabLayout] ⏱️ Waiting 50ms before restarting STT
|
||||||
|
[Android] [TabLayout] ⏰ Delay complete - restarting STT now
|
||||||
|
[Android] [TabLayout] ▶️ STARTING STT... (status: listening)
|
||||||
|
[Android] [TabLayout] ✅ STT started successfully ← КРИТИЧНО! Если нет этой строки — БАГ!
|
||||||
|
```
|
||||||
|
|
||||||
|
### Тест 2: Прерывание Julia голосом (только iOS)
|
||||||
|
|
||||||
|
**На iOS:**
|
||||||
|
1. Нажать FAB
|
||||||
|
2. Сказать: "Tell me a long story"
|
||||||
|
3. Пока Julia говорит, сказать: "Stop"
|
||||||
|
4. **Проверить:** Julia прервалась
|
||||||
|
5. **Проверить:** "Stop" отправилось как новый запрос
|
||||||
|
|
||||||
|
**На Android:**
|
||||||
|
Прерывание голосом **НЕ РАБОТАЕТ** (STT выключен во время TTS). Прервать можно только кнопкой FAB.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Checklist для отладки
|
||||||
|
|
||||||
|
Если STT не рестартует после TTS на Android:
|
||||||
|
|
||||||
|
- [ ] Проверь что `stopListening()` вызывается при `status === 'speaking'`
|
||||||
|
- [ ] Проверь что `sttIsListening === false` после `stopListening()`
|
||||||
|
- [ ] Проверь что `safeStartSTT()` вызывается после задержки 50ms
|
||||||
|
- [ ] Проверь что `sttStartingRef.current === false` перед вызовом `safeStartSTT()`
|
||||||
|
- [ ] Проверь что `sessionActiveRef.current === true` (сессия активна)
|
||||||
|
- [ ] Проверь логи на наличие `[Android] [TabLayout] ✅ STT started successfully`
|
||||||
|
- [ ] Проверь что нет логов `[Android] [TabLayout] ⚠️ SKIPPING STT start - TTS is playing`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Audio Focus на Android (детально)
|
||||||
|
|
||||||
|
### Как работает Audio Focus:
|
||||||
|
|
||||||
|
1. **TTS запрашивает Audio Focus:**
|
||||||
|
- Когда `Speech.speak()` вызывается, Android TTS запрашивает `AUDIOFOCUS_GAIN_TRANSIENT`
|
||||||
|
- Это **автоматически** останавливает все другие аудио-источники, включая STT
|
||||||
|
|
||||||
|
2. **STT теряет Audio Focus:**
|
||||||
|
- Expo Speech Recognition получает событие `onAudioFocusLoss`
|
||||||
|
- STT автоматически останавливается (не нужно вручную вызывать `stopListening()`)
|
||||||
|
|
||||||
|
3. **TTS освобождает Audio Focus:**
|
||||||
|
- Когда TTS заканчивает (`onDone` callback), Audio Focus освобождается
|
||||||
|
- Но STT **НЕ** рестартует автоматически — нужно вручную вызвать `startListening()`
|
||||||
|
|
||||||
|
### Почему 50ms delay на Android?
|
||||||
|
|
||||||
|
- TTS может освободить Audio Focus с небольшой задержкой
|
||||||
|
- Если попытаться запустить STT слишком рано, STT может не получить Audio Focus
|
||||||
|
- 50ms — достаточно для освобождения, но незаметно для юзера
|
||||||
|
|
||||||
|
### Почему 300ms delay на iOS?
|
||||||
|
|
||||||
|
- iOS не имеет проблем с Audio Focus
|
||||||
|
- 300ms delay нужен для плавного перехода анимаций (зелёный индикатор)
|
||||||
|
- Также даёт время для TTS полностью завершиться и освободить аудио-драйвер
|
||||||
@ -45,39 +45,55 @@ export default function TabLayout() {
|
|||||||
// so interruption on Android happens via FAB press instead.
|
// so interruption on Android happens via FAB press instead.
|
||||||
// On iOS, STT can run alongside TTS, so voice detection works.
|
// On iOS, STT can run alongside TTS, so voice detection works.
|
||||||
const handleVoiceDetected = useCallback(() => {
|
const handleVoiceDetected = useCallback(() => {
|
||||||
|
const platformPrefix = Platform.OS === 'ios' ? '[iOS]' : '[Android]';
|
||||||
|
console.log(`${platformPrefix} [TabLayout] handleVoiceDetected called - status: ${status}, isSpeaking: ${isSpeaking}`);
|
||||||
|
|
||||||
if (Platform.OS === 'ios' && (status === 'speaking' || isSpeaking)) {
|
if (Platform.OS === 'ios' && (status === 'speaking' || isSpeaking)) {
|
||||||
console.log('[TabLayout] Voice detected during TTS (iOS) - INTERRUPTING Julia');
|
console.log('[iOS] [TabLayout] Voice detected during TTS - INTERRUPTING Julia');
|
||||||
interruptIfSpeaking();
|
interruptIfSpeaking();
|
||||||
|
} else if (Platform.OS === 'android') {
|
||||||
|
console.log('[Android] [TabLayout] Voice detected but ignoring (STT disabled during TTS on Android)');
|
||||||
}
|
}
|
||||||
}, [status, isSpeaking, interruptIfSpeaking]);
|
}, [status, isSpeaking, interruptIfSpeaking]);
|
||||||
|
|
||||||
// Callback when STT ends - may need to restart if session is still active
|
// Callback when STT ends - may need to restart if session is still active
|
||||||
const handleSTTEnd = useCallback(() => {
|
const handleSTTEnd = useCallback(() => {
|
||||||
console.log('[TabLayout] STT ended, sessionActive:', sessionActiveRef.current);
|
const platformPrefix = Platform.OS === 'ios' ? '[iOS]' : '[Android]';
|
||||||
|
console.log(`${platformPrefix} [TabLayout] handleSTTEnd - sessionActive: ${sessionActiveRef.current}, status: ${status}`);
|
||||||
|
|
||||||
// If session is still active (user didn't stop it), we should restart STT
|
// If session is still active (user didn't stop it), we should restart STT
|
||||||
// This ensures STT continues during and after TTS playback
|
// This ensures STT continues during and after TTS playback
|
||||||
if (sessionActiveRef.current) {
|
if (sessionActiveRef.current) {
|
||||||
shouldRestartSTTRef.current = true;
|
shouldRestartSTTRef.current = true;
|
||||||
|
console.log(`${platformPrefix} [TabLayout] → shouldRestartSTT set to TRUE`);
|
||||||
|
} else {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] → Session not active, will NOT restart STT`);
|
||||||
}
|
}
|
||||||
}, []);
|
}, [status]);
|
||||||
|
|
||||||
// Callback for STT results
|
// Callback for STT results
|
||||||
const handleSpeechResult = useCallback((transcript: string, isFinal: boolean) => {
|
const handleSpeechResult = useCallback((transcript: string, isFinal: boolean) => {
|
||||||
|
const platformPrefix = Platform.OS === 'ios' ? '[iOS]' : '[Android]';
|
||||||
|
console.log(`${platformPrefix} [TabLayout] handleSpeechResult - isFinal: ${isFinal}, status: ${status}, transcript: "${transcript.slice(0, 40)}..."`);
|
||||||
|
|
||||||
// Ignore any STT results during TTS playback or processing (echo prevention)
|
// Ignore any STT results during TTS playback or processing (echo prevention)
|
||||||
if (status === 'speaking' || status === 'processing') {
|
if (status === 'speaking' || status === 'processing') {
|
||||||
if (isFinal) {
|
if (isFinal) {
|
||||||
// User interrupted Julia with speech — store to send after TTS stops
|
// User interrupted Julia with speech — store to send after TTS stops
|
||||||
console.log('[TabLayout] Got final result during TTS/processing - storing for after interruption:', transcript);
|
console.log(`${platformPrefix} [TabLayout] Got FINAL result during ${status} - storing for after interruption: "${transcript}"`);
|
||||||
pendingInterruptTranscriptRef.current = transcript;
|
pendingInterruptTranscriptRef.current = transcript;
|
||||||
|
} else {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] Ignoring PARTIAL transcript during ${status} (likely echo)`);
|
||||||
}
|
}
|
||||||
// Ignore partial transcripts during TTS (they're likely echo)
|
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (isFinal) {
|
if (isFinal) {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] → Processing FINAL transcript, sending to API`);
|
||||||
setTranscript(transcript);
|
setTranscript(transcript);
|
||||||
sendTranscript(transcript);
|
sendTranscript(transcript);
|
||||||
} else {
|
} else {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] → Updating PARTIAL transcript`);
|
||||||
setPartialTranscript(transcript);
|
setPartialTranscript(transcript);
|
||||||
}
|
}
|
||||||
}, [setTranscript, setPartialTranscript, sendTranscript, status]);
|
}, [setTranscript, setPartialTranscript, sendTranscript, status]);
|
||||||
@ -98,6 +114,8 @@ export default function TabLayout() {
|
|||||||
|
|
||||||
// Ref to prevent concurrent startListening calls
|
// Ref to prevent concurrent startListening calls
|
||||||
const sttStartingRef = useRef(false);
|
const sttStartingRef = useRef(false);
|
||||||
|
// Ref to ignore AppState changes during STT start (Android bug workaround)
|
||||||
|
const sttStartingIgnoreAppStateRef = useRef(false);
|
||||||
// Ref to track last partial transcript for iOS auto-stop
|
// Ref to track last partial transcript for iOS auto-stop
|
||||||
const lastPartialTextRef = useRef('');
|
const lastPartialTextRef = useRef('');
|
||||||
const silenceTimerRef = useRef<NodeJS.Timeout | null>(null);
|
const silenceTimerRef = useRef<NodeJS.Timeout | null>(null);
|
||||||
@ -105,6 +123,8 @@ export default function TabLayout() {
|
|||||||
// iOS AUTO-STOP: Stop STT after 2 seconds of silence (no new partial transcripts)
|
// iOS AUTO-STOP: Stop STT after 2 seconds of silence (no new partial transcripts)
|
||||||
// This triggers onEnd → iOS fix sends lastPartial as final
|
// This triggers onEnd → iOS fix sends lastPartial as final
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
|
const platformPrefix = Platform.OS === 'ios' ? '[iOS]' : '[Android]';
|
||||||
|
|
||||||
// Clear existing timer
|
// Clear existing timer
|
||||||
if (silenceTimerRef.current) {
|
if (silenceTimerRef.current) {
|
||||||
clearTimeout(silenceTimerRef.current);
|
clearTimeout(silenceTimerRef.current);
|
||||||
@ -118,15 +138,22 @@ export default function TabLayout() {
|
|||||||
|
|
||||||
// If partial changed, update ref and set new 2s timer
|
// If partial changed, update ref and set new 2s timer
|
||||||
if (currentPartial !== lastPartialTextRef.current) {
|
if (currentPartial !== lastPartialTextRef.current) {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] Partial changed: "${lastPartialTextRef.current}" → "${currentPartial}"`);
|
||||||
lastPartialTextRef.current = currentPartial;
|
lastPartialTextRef.current = currentPartial;
|
||||||
|
|
||||||
// Start 2-second silence timer
|
// Start 2-second silence timer
|
||||||
silenceTimerRef.current = setTimeout(() => {
|
silenceTimerRef.current = setTimeout(() => {
|
||||||
if (sttIsListening && sessionActiveRef.current) {
|
if (sttIsListening && sessionActiveRef.current) {
|
||||||
console.log('[TabLayout] 🍎 iOS AUTO-STOP: 2s silence - stopping STT to trigger onEnd → iOS fix');
|
if (Platform.OS === 'ios') {
|
||||||
|
console.log('[iOS] [TabLayout] 🍎 AUTO-STOP: 2s silence - stopping STT to trigger onEnd → iOS fix');
|
||||||
|
} else {
|
||||||
|
console.log('[Android] [TabLayout] 🤖 AUTO-STOP: 2s silence - stopping STT');
|
||||||
|
}
|
||||||
stopListening();
|
stopListening();
|
||||||
}
|
}
|
||||||
}, 2000);
|
}, 2000);
|
||||||
|
|
||||||
|
console.log(`${platformPrefix} [TabLayout] → Started 2s silence timer`);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -140,19 +167,44 @@ export default function TabLayout() {
|
|||||||
|
|
||||||
// Safe wrapper to start STT with debounce protection
|
// Safe wrapper to start STT with debounce protection
|
||||||
const safeStartSTT = useCallback(() => {
|
const safeStartSTT = useCallback(() => {
|
||||||
|
const platformPrefix = Platform.OS === 'ios' ? '[iOS]' : '[Android]';
|
||||||
|
|
||||||
if (sttIsListening || sttStartingRef.current) {
|
if (sttIsListening || sttStartingRef.current) {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] safeStartSTT - already listening or starting, skipping`);
|
||||||
return; // Already listening or starting
|
return; // Already listening or starting
|
||||||
}
|
}
|
||||||
|
|
||||||
// Don't start STT during TTS on Android - they share audio focus
|
// Don't start STT during TTS on Android - they share audio focus
|
||||||
if (Platform.OS === 'android' && (status === 'speaking' || isSpeaking)) {
|
if (Platform.OS === 'android' && (status === 'speaking' || isSpeaking)) {
|
||||||
console.log('[TabLayout] Skipping STT start - TTS is playing (Android audio focus)');
|
console.log('[Android] [TabLayout] ⚠️ SKIPPING STT start - TTS is playing (audio focus conflict)');
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
sttStartingRef.current = true;
|
sttStartingRef.current = true;
|
||||||
console.log('[TabLayout] Starting STT...');
|
|
||||||
startListening().finally(() => {
|
// ANDROID BUG WORKAROUND: startListening() triggers AppState change to background
|
||||||
sttStartingRef.current = false;
|
// Ignore AppState changes for 200ms after starting STT
|
||||||
});
|
if (Platform.OS === 'android') {
|
||||||
|
sttStartingIgnoreAppStateRef.current = true;
|
||||||
|
console.log('[Android] [TabLayout] 🛡️ Ignoring AppState changes for 200ms (STT start workaround)');
|
||||||
|
setTimeout(() => {
|
||||||
|
sttStartingIgnoreAppStateRef.current = false;
|
||||||
|
console.log('[Android] [TabLayout] ✅ AppState monitoring resumed');
|
||||||
|
}, 200);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`${platformPrefix} [TabLayout] ▶️ STARTING STT... (status: ${status})`);
|
||||||
|
|
||||||
|
startListening()
|
||||||
|
.then(() => {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] ✅ STT started successfully`);
|
||||||
|
})
|
||||||
|
.catch((err) => {
|
||||||
|
console.error(`${platformPrefix} [TabLayout] ❌ STT start failed:`, err);
|
||||||
|
})
|
||||||
|
.finally(() => {
|
||||||
|
sttStartingRef.current = false;
|
||||||
|
});
|
||||||
}, [sttIsListening, status, isSpeaking, startListening]);
|
}, [sttIsListening, status, isSpeaking, startListening]);
|
||||||
|
|
||||||
// Update session active ref when isListening changes
|
// Update session active ref when isListening changes
|
||||||
@ -165,11 +217,13 @@ export default function TabLayout() {
|
|||||||
|
|
||||||
// Start/stop STT when voice session starts/stops
|
// Start/stop STT when voice session starts/stops
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
|
const platformPrefix = Platform.OS === 'ios' ? '[iOS]' : '[Android]';
|
||||||
|
|
||||||
if (isListening) {
|
if (isListening) {
|
||||||
console.log('[TabLayout] Voice session started - starting STT');
|
console.log(`${platformPrefix} [TabLayout] 🎤 Voice session STARTED - starting STT`);
|
||||||
safeStartSTT();
|
safeStartSTT();
|
||||||
} else {
|
} else {
|
||||||
console.log('[TabLayout] Voice session ended - stopping STT');
|
console.log(`${platformPrefix} [TabLayout] 🛑 Voice session ENDED - stopping STT`);
|
||||||
stopListening();
|
stopListening();
|
||||||
}
|
}
|
||||||
}, [isListening]); // eslint-disable-line react-hooks/exhaustive-deps
|
}, [isListening]); // eslint-disable-line react-hooks/exhaustive-deps
|
||||||
@ -180,23 +234,26 @@ export default function TabLayout() {
|
|||||||
// Stop STT when entering processing or speaking state (prevent echo)
|
// Stop STT when entering processing or speaking state (prevent echo)
|
||||||
// Restart STT when TTS finishes (speaking → listening)
|
// Restart STT when TTS finishes (speaking → listening)
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
|
const platformPrefix = Platform.OS === 'ios' ? '[iOS]' : '[Android]';
|
||||||
const prevStatus = prevStatusRef.current;
|
const prevStatus = prevStatusRef.current;
|
||||||
prevStatusRef.current = status;
|
prevStatusRef.current = status;
|
||||||
|
|
||||||
|
console.log(`${platformPrefix} [TabLayout] Status transition: ${prevStatus} → ${status}, sttIsListening: ${sttIsListening}`);
|
||||||
|
|
||||||
// Stop STT when processing starts or TTS starts (prevent Julia hearing herself)
|
// Stop STT when processing starts or TTS starts (prevent Julia hearing herself)
|
||||||
if ((status === 'processing' || status === 'speaking') && sttIsListening) {
|
if ((status === 'processing' || status === 'speaking') && sttIsListening) {
|
||||||
console.log('[TabLayout] Stopping STT during', status, '(echo prevention)');
|
console.log(`${platformPrefix} [TabLayout] ⏸️ Stopping STT during ${status} (echo prevention)`);
|
||||||
stopListening();
|
stopListening();
|
||||||
}
|
}
|
||||||
|
|
||||||
// When TTS finishes (speaking → listening), restart STT
|
// When TTS finishes (speaking → listening), restart STT
|
||||||
if (prevStatus === 'speaking' && status === 'listening' && sessionActiveRef.current) {
|
if (prevStatus === 'speaking' && status === 'listening' && sessionActiveRef.current) {
|
||||||
console.log('[TabLayout] TTS finished - restarting STT');
|
console.log(`${platformPrefix} [TabLayout] 🔄 TTS FINISHED - preparing to restart STT`);
|
||||||
|
|
||||||
// Process pending transcript from interruption if any
|
// Process pending transcript from interruption if any
|
||||||
const pendingTranscript = pendingInterruptTranscriptRef.current;
|
const pendingTranscript = pendingInterruptTranscriptRef.current;
|
||||||
if (pendingTranscript) {
|
if (pendingTranscript) {
|
||||||
console.log('[TabLayout] Processing pending interrupt transcript:', pendingTranscript);
|
console.log(`${platformPrefix} [TabLayout] 📝 Processing pending interrupt transcript: "${pendingTranscript}"`);
|
||||||
pendingInterruptTranscriptRef.current = null;
|
pendingInterruptTranscriptRef.current = null;
|
||||||
setTranscript(pendingTranscript);
|
setTranscript(pendingTranscript);
|
||||||
sendTranscript(pendingTranscript);
|
sendTranscript(pendingTranscript);
|
||||||
@ -206,9 +263,14 @@ export default function TabLayout() {
|
|||||||
// iOS: 300ms for smooth audio fade
|
// iOS: 300ms for smooth audio fade
|
||||||
// Android: 50ms (Audio Focus releases immediately)
|
// Android: 50ms (Audio Focus releases immediately)
|
||||||
const delay = Platform.OS === 'android' ? 50 : 300;
|
const delay = Platform.OS === 'android' ? 50 : 300;
|
||||||
|
console.log(`${platformPrefix} [TabLayout] ⏱️ Waiting ${delay}ms before restarting STT (audio focus release)`);
|
||||||
|
|
||||||
const timer = setTimeout(() => {
|
const timer = setTimeout(() => {
|
||||||
if (sessionActiveRef.current) {
|
if (sessionActiveRef.current) {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] ⏰ Delay complete - restarting STT now`);
|
||||||
safeStartSTT();
|
safeStartSTT();
|
||||||
|
} else {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] ⚠️ Session stopped during delay, NOT restarting STT`);
|
||||||
}
|
}
|
||||||
}, delay);
|
}, delay);
|
||||||
return () => clearTimeout(timer);
|
return () => clearTimeout(timer);
|
||||||
@ -220,6 +282,8 @@ export default function TabLayout() {
|
|||||||
|
|
||||||
// When STT ends unexpectedly during active session, restart it (but not during TTS)
|
// When STT ends unexpectedly during active session, restart it (but not during TTS)
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
|
const platformPrefix = Platform.OS === 'ios' ? '[iOS]' : '[Android]';
|
||||||
|
|
||||||
if (
|
if (
|
||||||
shouldRestartSTTRef.current &&
|
shouldRestartSTTRef.current &&
|
||||||
sessionActiveRef.current &&
|
sessionActiveRef.current &&
|
||||||
@ -228,10 +292,15 @@ export default function TabLayout() {
|
|||||||
status !== 'speaking'
|
status !== 'speaking'
|
||||||
) {
|
) {
|
||||||
shouldRestartSTTRef.current = false;
|
shouldRestartSTTRef.current = false;
|
||||||
console.log('[TabLayout] STT ended unexpectedly - restarting');
|
console.log(`${platformPrefix} [TabLayout] 🔄 STT ended UNEXPECTEDLY - will restart in 300ms`);
|
||||||
|
console.log(`${platformPrefix} [TabLayout] → Conditions: sessionActive=${sessionActiveRef.current}, status=${status}`);
|
||||||
|
|
||||||
const timer = setTimeout(() => {
|
const timer = setTimeout(() => {
|
||||||
if (sessionActiveRef.current) {
|
if (sessionActiveRef.current) {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] ⏰ Restarting STT after unexpected end`);
|
||||||
safeStartSTT();
|
safeStartSTT();
|
||||||
|
} else {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] ⚠️ Session stopped during delay, NOT restarting`);
|
||||||
}
|
}
|
||||||
}, 300);
|
}, 300);
|
||||||
return () => clearTimeout(timer);
|
return () => clearTimeout(timer);
|
||||||
@ -241,10 +310,20 @@ export default function TabLayout() {
|
|||||||
// Handle app state changes (background/foreground)
|
// Handle app state changes (background/foreground)
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
const handleAppStateChange = (nextAppState: AppStateStatus) => {
|
const handleAppStateChange = (nextAppState: AppStateStatus) => {
|
||||||
|
const platformPrefix = Platform.OS === 'ios' ? '[iOS]' : '[Android]';
|
||||||
|
console.log(`${platformPrefix} [TabLayout] 📱 AppState changed to: "${nextAppState}"`);
|
||||||
|
|
||||||
|
// ANDROID BUG WORKAROUND: Ignore AppState changes during STT start
|
||||||
|
// startListening() triggers spurious background transition on Android
|
||||||
|
if (Platform.OS === 'android' && sttStartingIgnoreAppStateRef.current) {
|
||||||
|
console.log(`[Android] [TabLayout] 🛡️ IGNORING AppState change (STT start protection active)`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
// When app goes to background/inactive - stop voice session
|
// When app goes to background/inactive - stop voice session
|
||||||
// STT/TTS cannot work in background, so it's pointless to keep session active
|
// STT/TTS cannot work in background, so it's pointless to keep session active
|
||||||
if ((nextAppState === 'background' || nextAppState === 'inactive') && sessionActiveRef.current) {
|
if ((nextAppState === 'background' || nextAppState === 'inactive') && sessionActiveRef.current) {
|
||||||
console.log('[TabLayout] App going to background - stopping voice session');
|
console.log(`${platformPrefix} [TabLayout] App going to ${nextAppState} - stopping voice session`);
|
||||||
stopListening();
|
stopListening();
|
||||||
stopSession();
|
stopSession();
|
||||||
sessionActiveRef.current = false;
|
sessionActiveRef.current = false;
|
||||||
@ -255,7 +334,7 @@ export default function TabLayout() {
|
|||||||
// When app comes back to foreground - do NOT auto-restart session
|
// When app comes back to foreground - do NOT auto-restart session
|
||||||
// User must manually press FAB to start new session
|
// User must manually press FAB to start new session
|
||||||
if (nextAppState === 'active') {
|
if (nextAppState === 'active') {
|
||||||
console.log('[TabLayout] App foregrounded - session remains stopped (user must restart via FAB)');
|
console.log(`${platformPrefix} [TabLayout] App foregrounded - session remains stopped (user must restart via FAB)`);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
@ -266,18 +345,21 @@ export default function TabLayout() {
|
|||||||
// Handle voice FAB press - toggle listening mode
|
// Handle voice FAB press - toggle listening mode
|
||||||
// Must check ALL active states (listening, processing, speaking), not just isListening
|
// Must check ALL active states (listening, processing, speaking), not just isListening
|
||||||
const handleVoiceFABPress = useCallback(() => {
|
const handleVoiceFABPress = useCallback(() => {
|
||||||
|
const platformPrefix = Platform.OS === 'ios' ? '[iOS]' : '[Android]';
|
||||||
const isSessionActive = isListening || status === 'speaking' || status === 'processing';
|
const isSessionActive = isListening || status === 'speaking' || status === 'processing';
|
||||||
console.log('[TabLayout] FAB pressed, isSessionActive:', isSessionActive, 'status:', status, 'isListening:', isListening);
|
console.log(`${platformPrefix} [TabLayout] 🎯 FAB PRESSED - isSessionActive: ${isSessionActive}, status: ${status}, isListening: ${isListening}`);
|
||||||
|
|
||||||
if (isSessionActive) {
|
if (isSessionActive) {
|
||||||
// Force-stop everything: STT, TTS, and session state
|
// Force-stop everything: STT, TTS, and session state
|
||||||
console.log('[TabLayout] Force-stopping everything');
|
console.log(`${platformPrefix} [TabLayout] 🛑 FORCE-STOPPING everything (FAB stop)`);
|
||||||
stopListening();
|
stopListening();
|
||||||
stopSession();
|
stopSession();
|
||||||
sessionActiveRef.current = false;
|
sessionActiveRef.current = false;
|
||||||
shouldRestartSTTRef.current = false;
|
shouldRestartSTTRef.current = false;
|
||||||
pendingInterruptTranscriptRef.current = null;
|
pendingInterruptTranscriptRef.current = null;
|
||||||
|
console.log(`${platformPrefix} [TabLayout] → All flags cleared`);
|
||||||
} else {
|
} else {
|
||||||
|
console.log(`${platformPrefix} [TabLayout] ▶️ STARTING session (FAB start)`);
|
||||||
startSession();
|
startSession();
|
||||||
}
|
}
|
||||||
}, [isListening, status, startSession, stopSession, stopListening]);
|
}, [isListening, status, startSession, stopSession, stopListening]);
|
||||||
@ -351,11 +433,14 @@ export default function TabLayout() {
|
|||||||
),
|
),
|
||||||
}}
|
}}
|
||||||
/>
|
/>
|
||||||
{/* Voice Debug - hidden from tab bar */}
|
{/* Voice Debug - VISIBLE for debugging */}
|
||||||
<Tabs.Screen
|
<Tabs.Screen
|
||||||
name="voice-debug"
|
name="voice-debug"
|
||||||
options={{
|
options={{
|
||||||
href: null,
|
title: 'Debug',
|
||||||
|
tabBarIcon: ({ color, size }) => (
|
||||||
|
<Feather name="activity" size={22} color={color} />
|
||||||
|
),
|
||||||
}}
|
}}
|
||||||
/>
|
/>
|
||||||
<Tabs.Screen
|
<Tabs.Screen
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user