mirror of
https://github.com/souravanand001/ai-assistant-android.git
synced 2026-10-19 11:04:18 +00:00
updated README
This commit is contained in:
souravanand001
parent
a1e0001e23
commit
f2b2bb9bfe
5 changed files with 196 additions and 227 deletions
423
README.md
423
README.md
|
|
@ -1,5 +1,3 @@
|
|||
<div align="center">
|
||||
|
||||
# AI Assistant for Android
|
||||
|
||||
[](LICENSE)
|
||||
|
|
@ -9,121 +7,105 @@
|
|||
[](https://www.tensorflow.org/lite)
|
||||
[](https://github.com/k2-fsa/sherpa-onnx)
|
||||
|
||||
**A private, offline-first alternative to Gemini Live with real-time screen vision, encrypted storage, and Bring-Your-Own-Key (BYOK) LLM support, engineered to replace the system default assistant.**
|
||||
An offline-first Android assistant that runs locally on your device. It supports screen vision, live camera sharing with real-time voice chat, encrypted data storage, and lets you connect your own LLM API keys. You can set it as your default phone assistant.
|
||||
|
||||
<div align="center">
|
||||
|
||||
<p>
|
||||
<img src="screenshots/gif1.gif" alt="App Overview" width="300">
|
||||
</p>
|
||||
|
||||
<table>
|
||||
<tr align="center">
|
||||
<th width="33%">Conversational Chat UI</th>
|
||||
<th width="33%">Hands Free Floating Orb</th>
|
||||
<th width="33%">System Settings</th>
|
||||
</tr>
|
||||
<tr align="center">
|
||||
<td width="33%"><img src="screenshots/image1.png" width="100%"></td>
|
||||
<td width="33%"><img src="screenshots/image2.png" width="100%"></td>
|
||||
<td width="33%"><img src="screenshots/image3.png" width="100%"></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 📋 Table of Contents
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#-overview)
|
||||
- [Why This Project?](#-why-this-project)
|
||||
- [Features](#-features)
|
||||
- [Screenshots](#-screenshots)
|
||||
- [Architecture Overview](#%EF%B8%8F-architecture-overview)
|
||||
- [Request Processing Flow](#-request-processing-flow)
|
||||
- [Technologies Used](#-technologies-used)
|
||||
- [Project Structure](#-project-structure)
|
||||
- [Setup](#-setup)
|
||||
- [API Configuration](#-api-configuration)
|
||||
- [Permissions](#-permissions)
|
||||
- [Supported Features Table](#-supported-features-table)
|
||||
- [Privacy & Security](#-privacy--security)
|
||||
- [Roadmap](#-roadmap)
|
||||
- [Contributing](#-contributing)
|
||||
- [License](#-license)
|
||||
- [Overview](#overview)
|
||||
- [Features](#features)
|
||||
- [Architecture Overview](#architecture-overview)
|
||||
- [Request Processing Flow](#request-processing-flow)
|
||||
- [Technologies Used](#technologies-used)
|
||||
- [Project Structure](#project-structure)
|
||||
- [Setup](#setup)
|
||||
- [API Configuration](#api-configuration)
|
||||
- [Permissions](#permissions)
|
||||
- [Supported Features Table](#supported-features-table)
|
||||
- [Privacy & Security](#privacy--security)
|
||||
- [Known Issues](#known-issues)
|
||||
- [Roadmap](#roadmap)
|
||||
- [Contributing](#contributing)
|
||||
- [License](#license)
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Overview
|
||||
## Overview
|
||||
|
||||
AI Assistant is a production-quality Android voice assistant designed around three principles: **privacy**, **offline capability**, and **extensibility**. It integrates a custom three-layer audio pipeline, on-device intent classification, encrypted conversation storage, real-time screen vision, and a flexible LLM adapter that works with any OpenAI-compatible API endpoint.
|
||||
This app is a private voice assistant for Android. It runs offline as much as possible, encrypts your conversations, and reads your screen when asked. You can connect it to any OpenAI-compatible LLM endpoint.
|
||||
|
||||
Unlike consumer assistants, no voice data is sent to a vendor. The app hooks directly into Android's system assistant slot, giving it the same home-button activation as Google Assistant — but with full control over what happens with your data.
|
||||
Unlike standard assistants, it does not send your voice data to external servers. It replaces Android's default assistant, so you can launch it with a long press on the home button.
|
||||
|
||||
---
|
||||
|
||||
## 💡 Why This Project?
|
||||
## Features
|
||||
|
||||
Most Android assistant apps are thin wrappers around a cloud API. This project takes a fundamentally different approach:
|
||||
### AI & LLM
|
||||
- **Custom LLM Providers:** Works with Groq, OpenAI, Anthropic, Gemini, Ollama, LM Studio, etc.
|
||||
- **Vision Auto-Detection:** Automatically tests if your model supports images and enables screen vision and live camera features if it does.
|
||||
- **Fast Speech Output:** Starts speaking sentences before the model finishes generating the full response.
|
||||
|
||||
| Typical Assistant App | This Project |
|
||||
|---|---|
|
||||
| Cloud-only speech recognition | On-device VAD + selectable STT engines |
|
||||
| No conversation encryption | AES-256-GCM via Android KeyStore |
|
||||
| Single fixed LLM provider | Plug any OpenAI/Gemini/Anthropic-compatible endpoint |
|
||||
| No screen awareness | Real-time screen capture with temporal frame syncing |
|
||||
| Hands-free requires wake word cloud service | On-device Silero VAD with barge-in interruption
|
||||
### Voice Pipeline
|
||||
- **Voice Activation (VAD):** Detects when you start and stop talking locally on your device.
|
||||
- **Interruption Support:** You can speak over the assistant to interrupt it.
|
||||
- **Hands-Free Mode:** Automatically listens for your next response after speaking.
|
||||
- **Audio Routing:** Supports Bluetooth headsets and speakerphone.
|
||||
|
||||
The result is an assistant that can run mostly offline, understands what's on your screen, and stores nothing in plaintext.
|
||||
### Screen Vision
|
||||
- **Screen Capture:** Captures screen frames locally in the background.
|
||||
- **Smart Frame Syncing:** Matches the captured frame to the exact moment you started speaking.
|
||||
- **Camera Sharing:** Share your live camera feed to discuss your real-world surroundings with the assistant.
|
||||
- **Privacy Controls:** Easily pause or stop screen capture from a system notification.
|
||||
|
||||
### Privacy & Encryption
|
||||
- **Secure Storage:** Encrypts conversations and attachments using AES-256-GCM.
|
||||
- **Safe Deletion:** Securely deletes files from the device when you delete a message or chat.
|
||||
|
||||
### Offline Tools
|
||||
- **Offline Actions:** Processes commands (like calling or navigation) offline using an on-device model.
|
||||
- **Offline Speech & Voice:** Handles speech-to-text and text-to-speech without internet.
|
||||
- **Offline Translation:** Translates input to English locally using Google ML Kit.
|
||||
|
||||
### Android Integration
|
||||
- **Default Assistant:** Launch by long-pressing the home button or swiping.
|
||||
- **Lock Screen Support:** Answer voice queries without unlocking your phone.
|
||||
- **System Actions:** Set alarms, dial contacts (with fuzzy name matching), search YouTube music, and check the weather.
|
||||
|
||||
---
|
||||
|
||||
## ✨ Features
|
||||
|
||||
### 🤖 AI & LLM
|
||||
- Connect to **any REST LLM endpoint** — Groq, OpenAI, Anthropic, Gemini, Ollama, LM Studio
|
||||
- Automatic **capability probing**: sends a test image to detect vision support and dynamically enables multimodal features
|
||||
- **Streaming token playback**: first sentence starts speaking before the LLM finishes generating
|
||||
## Architecture Overview
|
||||
|
||||
### 🎙️ Voice Pipeline
|
||||
- **Three-layer audio architecture**: hardware echo cancellation → on-device VAD → state machine orchestrator
|
||||
- **Silero VAD** (via Sherpa-ONNX) for precise speech boundary detection, no cloud round-trips
|
||||
- **Barge-in / interruption**: speak while the assistant is talking to interrupt it with a 300 ms echo-bleed guard
|
||||
- **Hands-free auto-loop**: automatically re-enters listening state after each assistant response
|
||||
- Bluetooth SCO and Speakerphone routing for headsets and hands-free modes
|
||||
|
||||
### 👁️ Screen Vision
|
||||
- Background **screen capture service** at ~6 FPS into a bounded circular frame buffer
|
||||
- **Temporal frame syncing**: retrieves the frame closest to the moment speech started — so the LLM sees exactly what you were looking at
|
||||
- Frames scaled to 720 px and JPEG-compressed before being sent to the LLM
|
||||
- Controllable via a persistent Material You notification with Stop and Mute actions
|
||||
|
||||
### 🔐 Privacy & Security
|
||||
- All conversation text encrypted with **AES-256-GCM** via a hardware-backed Android KeyStore key
|
||||
- Unique IV generated per database entry; no two records share the same nonce
|
||||
- Attached images, documents, and captured frames encrypted before writing to disk
|
||||
- **Wipe-clean deletion**: files decrypted, dereferenced, and removed from disk on message/chat deletion
|
||||
|
||||
### 🌐 Offline Intelligence
|
||||
- On-device **intent classification** using a custom 26 MB TFLite model via MediaPipe
|
||||
- On-device **speech-to-text** option via Parakeet (Sherpa-ONNX)
|
||||
- On-device **TTS** using a VITS Supertonic model (Sherpa-ONNX), no API key required
|
||||
- Bidirectional **offline translation** for non-English users via Google ML Kit language packs
|
||||
|
||||
### 📱 Android Integration
|
||||
- Replaces the **system default assistant** (home button long-press, diagonal swipe)
|
||||
- **Lock screen bypass**: responds to voice queries without unlocking the device
|
||||
- Fuzzy **contact matching** (exact → token → prefix similarity) before dialing
|
||||
- Native **alarm & reminder scheduling** with natural language time parsing
|
||||
- **Google Maps navigation** and **weather forecasts** via GPS Fused Location Provider
|
||||
- **YouTube music search** and playback
|
||||
|
||||
---
|
||||
|
||||
## 📸 Screenshots
|
||||
|
||||
| Conversational Chat UI | Hands-Free Floating Orb | System Settings |
|
||||
|:---:|:---:|:---:|
|
||||
|  |  |  |
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture Overview
|
||||
|
||||
The app is built around offline-first, modular components. Each layer can be swapped independently.
|
||||
The app is built around offline first, modular components. Each layer can be swapped independently.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[User — Voice / Text / File] --> B[MainViewModel]
|
||||
|
||||
subgraph Input Processing
|
||||
B --> C[TranslatorManager\nML Kit — Offline]
|
||||
B --> C[TranslatorManager\nML Kit Offline]
|
||||
C --> D[TextClassifierHelper\nMediaPipe TFLite]
|
||||
end
|
||||
|
||||
|
|
@ -164,88 +146,79 @@ graph TD
|
|||
end
|
||||
```
|
||||
|
||||
**Key design decisions:**
|
||||
|
||||
- **No hardcoded LLM**: `FlexibleLlmAdapter` uses template substitution (`{{MODEL}}`, `{{MESSAGES}}`, `{{SYSTEM_CONTEXT}}`), making any REST endpoint a drop-in.
|
||||
- **Lexical safeguards**: ML classifiers can misfire. A vocabulary cross-check gates every intent before executing native actions.
|
||||
- **Encryption at rest, not in transit**: the app trusts the OS network stack for TLS but never stores anything in plaintext on disk.
|
||||
**Design details:**
|
||||
- **Flexible LLM Adapter:** Uses templates to support any OpenAI-compatible API endpoint.
|
||||
- **Intent Verification:** Checks user queries against keyword lists to prevent incorrect system actions.
|
||||
- **Local Encryption:** All conversation history and files are encrypted before saving to disk.
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Request Processing Flow
|
||||
## Request Processing Flow
|
||||
|
||||
<details>
|
||||
<summary><strong>Voice / Text Input</strong></summary>
|
||||
|
||||
The user speaks or types in the `ChatScreen`. Voice input flows through the three-layer audio pipeline:
|
||||
|
||||
1. `AudioHygieneProcessor` records at 16 kHz Mono 16-bit PCM and applies hardware AEC, Noise Suppression, and AGC.
|
||||
2. `VadIntelligenceProcessor` feeds PCM samples into Silero VAD in 512-sample windows to detect speech start and end boundaries.
|
||||
3. `VoiceStateMachine` tracks the conversation turn state and handles barge-in: if the user speaks while the assistant is responding, TTS is halted after a 300 ms echo-bleed window and listening resumes.
|
||||
When you speak or type, the app processes the audio locally:
|
||||
1. It records and cleans up the audio.
|
||||
2. It detects when you start and stop speaking.
|
||||
3. If you speak while the assistant is answering, it stops talking and listens to you.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Audio Processing & STT</strong></summary>
|
||||
|
||||
`SpeechRecognizerManager` routes recognized audio to the configured STT engine — Android's native recognizer, on-device Parakeet (Sherpa-ONNX), a hybrid mode, or a cloud API. The recognized text is passed to the `MainViewModel`.
|
||||
The app converts your speech to text using either Android's built-in system, an offline voice model, or a cloud API.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Translation & Classification</strong></summary>
|
||||
|
||||
If the user's language is set to non-English, `TranslatorManager` translates the input to English using an offline ML Kit language pack.
|
||||
|
||||
The English text is then passed to `TextClassifierHelper`, which runs the 26 MB custom TFLite model via MediaPipe to produce an intent label: `CALL`, `SONGS`, `ALARM`, `REMINDER`, `NAVIGATION`, `WEATHER`, `SETTINGS`, or `OTHER`.
|
||||
|
||||
`ProcessChatCommandUseCase` cross-checks the label against category-specific vocabulary maps. If no matching keywords are found, the intent is demoted to `OTHER`.
|
||||
1. If you speak a non-English language, the app translates it to English offline using Google ML Kit.
|
||||
2. It classifies the text to determine your intent (e.g., call, alarm, navigation, or general question).
|
||||
3. It double-checks keywords to make sure it doesn't run a system command by mistake.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Native Actions & Lock States</strong></summary>
|
||||
|
||||
Confirmed intents trigger native Android actions:
|
||||
When the app identifies a command, it runs the matching phone action:
|
||||
- **Calls:** Searches contacts and dials the number.
|
||||
- **Music:** Searches and plays songs on YouTube.
|
||||
- **Alarms/Reminders:** Sets system alarms using natural time phrasing.
|
||||
- **Navigation:** Launches Google Maps navigation.
|
||||
- **Weather:** Gets your location and displays the forecast.
|
||||
|
||||
- **Call** — fuzzy contact search (exact → token → prefix similarity > 0.7), then `CALL_PHONE` intent.
|
||||
- **Songs** — YouTube Data API v3 search, then playback.
|
||||
- **Alarm / Reminder** — natural language time parsing via regex, then system alarm scheduling.
|
||||
- **Navigation** — location extraction and `google.navigation:q=…` intent.
|
||||
- **Weather** — GMS Fused Location Provider → forecast fetch → weather card UI.
|
||||
|
||||
If an intent is confirmed but a required detail is missing (e.g., no contact name), a `LockState` is set. The next user turn bypasses classification and is routed directly to fill the missing variable, then the action fires.
|
||||
If a command is missing information (like a name for a call), the app will ask you for it directly on the next turn.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>LLM & Vision</strong></summary>
|
||||
|
||||
`OTHER` queries — or any query with an attached file — are sent to the configured LLM via `FlexibleLlmAdapter`. If screen vision is active, the frame closest to the moment speech began is retrieved from `VisualBufferManager` and attached to the request. Full conversation history is included as context.
|
||||
|
||||
`ModelCapabilityProber` previously verified whether the endpoint accepts image inputs; vision attachment is skipped if the model does not support it.
|
||||
General questions go to your selected LLM. If you use screen vision or live camera sharing, the app attaches the latest captured frame from your screen or camera when you start speaking, provided your selected LLM supports images.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Streaming TTS</strong></summary>
|
||||
|
||||
As the LLM streams response tokens, `MainViewModel` appends them to a text buffer and calls `findSentenceBoundary()` after each token. The parser uses `isAbbreviationOrDecimal()` to skip false boundaries like `Mr.`, `3.14`, or `e.g.`. The moment a complete sentence is detected, it is dequeued and sent to the active `TtsManager` — Edge TTS, Google TTS, Offline VITS, or Native — so speech begins before the LLM finishes generating.
|
||||
|
||||
After TTS completes in hands-free mode, `VoiceStateMachine` automatically returns to `LISTENING`.
|
||||
As the LLM generates a response, the app parses it sentence-by-sentence. Once a sentence is ready, the voice engine starts speaking it immediately so you don't have to wait for the full response to load.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Conversation Storage</strong></summary>
|
||||
|
||||
Every message — user and assistant — is encrypted with AES-256-GCM before writing to the Room database. A unique IV is generated per entry. Attached files are encrypted before writing to private app storage. On deletion, the repository decrypts file paths, removes the files from disk, and removes the database records.
|
||||
All messages are encrypted and saved locally. Attached files are also encrypted before saving. Deleting a message or chat deletes all associated files permanently.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Technologies Used
|
||||
## Technologies Used
|
||||
|
||||
| Category | Technology |
|
||||
|---|---|
|
||||
|
|
@ -262,204 +235,200 @@ Every message — user and assistant — is encrypted with AES-256-GCM before wr
|
|||
|
||||
---
|
||||
|
||||
## 📁 Project Structure
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
app/src/main/java/com/app/assistant/
|
||||
│
|
||||
├── AssistActivity.kt # System assistant entry point (ACTION_ASSIST hook)
|
||||
├── MainActivity.kt # UI host, permission manager, lifecycle observer
|
||||
├── AssistActivity.kt # Entry point for system assistant integration
|
||||
├── MainActivity.kt # Main UI activity and lifecycle management
|
||||
│
|
||||
├── api/ # Retrofit interfaces and HTTP clients
|
||||
├── api/ # HTTP clients and API services
|
||||
│
|
||||
├── camera/
|
||||
│ ├── ScreenCaptureService.kt # Foreground MediaProjection capture loop (~6 FPS)
|
||||
│ └── VisualBufferManager.kt # Bounded circular frame buffer with timestamp lookup
|
||||
│ ├── ScreenCaptureService.kt # Captures screenshots periodically
|
||||
│ └── VisualBufferManager.kt # Stores captured frames with timestamps
|
||||
│
|
||||
├── classifier/
|
||||
│ └── TextClassifierHelper.kt # MediaPipe TFLite intent classifier (model.tflite, 26 MB)
|
||||
│ └── TextClassifierHelper.kt # Local TFLite intent classifier
|
||||
│
|
||||
├── db/
|
||||
│ ├── AppDatabase.kt # Room database definition
|
||||
│ ├── EncryptionUtil.kt # AES-256-GCM + Android KeyStore key management
|
||||
│ ├── DynamicConversationRepo # CRUD with transparent encrypt/decrypt layer
|
||||
│ └── *Entity.kt # Schemas: Messages, Groups, Attachments
|
||||
│ ├── AppDatabase.kt # Room database setup
|
||||
│ ├── EncryptionUtil.kt # Encryption helpers (AES-GCM + KeyStore)
|
||||
│ ├── DynamicConversationRepo # Handles encrypted database operations
|
||||
│ └── *Entity.kt # Database schemas
|
||||
│
|
||||
├── llm/
|
||||
│ ├── FlexibleLlmAdapter.kt # Template-based HTTP LLM client (any REST endpoint)
|
||||
│ ├── ModelCapabilityProber.kt # Vision & model-list capability prober
|
||||
│ └── LlmMessage.kt # LLM request/response schemas
|
||||
│ ├── FlexibleLlmAdapter.kt # API adapter for various LLMs
|
||||
│ ├── ModelCapabilityProber.kt # Detects model vision capability
|
||||
│ └── LlmMessage.kt # LLM data schemas
|
||||
│
|
||||
├── repository/
|
||||
│ ├── ContactsRepository.kt # Offline fuzzy contact name matching
|
||||
│ ├── SettingsRepository.kt # SharedPreferences configuration wrapper
|
||||
│ └── WeatherRepository.kt # GPS geocoding and forecast fetcher
|
||||
│ ├── ContactsRepository.kt # Matches contact names offline
|
||||
│ ├── SettingsRepository.kt # App settings manager
|
||||
│ └── WeatherRepository.kt # Location and weather logic
|
||||
│
|
||||
├── speech/
|
||||
│ ├── AudioHygieneProcessor.kt # 16 kHz PCM recorder with AEC, NS, AGC
|
||||
│ ├── VadIntelligenceProcessor.kt # Silero VAD frame evaluator (512-sample windows)
|
||||
│ ├── VoiceStateMachine.kt # Turn lifecycle + barge-in logic
|
||||
│ └── SpeechRecognizerManager.kt # STT engine orchestrator
|
||||
│ ├── AudioHygieneProcessor.kt # Audio recording and processing
|
||||
│ ├── VadIntelligenceProcessor.kt # On-device speech detection (VAD)
|
||||
│ ├── VoiceStateMachine.kt # Manages voice states (listening, speaking)
|
||||
│ └── SpeechRecognizerManager.kt # Manages speech-to-text engines
|
||||
│
|
||||
├── translation/
|
||||
│ └── TranslatorManager.kt # ML Kit offline translation pack controller
|
||||
│ └── TranslatorManager.kt # Manages offline translation
|
||||
│
|
||||
├── tts/
|
||||
│ ├── TtsEngineSelector.kt # Selects and switches between TTS engines
|
||||
│ ├── EdgeTtsApiManager.kt # Microsoft Edge WebSocket neural TTS (free, no key)
|
||||
│ ├── GoogleTtsApiManager.kt # Google Cloud TTS via MediaCodec + AudioTrack
|
||||
│ ├── OfflineTtsManager.kt # Sherpa-ONNX VITS model (fully offline)
|
||||
│ └── NativeTtsManager.kt # Android system TextToSpeech fallback
|
||||
│ ├── TtsEngineSelector.kt # Switches between TTS engines
|
||||
│ ├── EdgeTtsApiManager.kt # Microsoft Edge TTS integration
|
||||
│ ├── GoogleTtsApiManager.kt # Google Cloud TTS integration
|
||||
│ ├── OfflineTtsManager.kt # On-device TTS integration
|
||||
│ └── NativeTtsManager.kt # Default system TTS fallback
|
||||
│
|
||||
└── ui/
|
||||
├── screen/
|
||||
│ ├── ChatScreen.kt # Main chat controller
|
||||
│ ├── ChatLayout.kt # Compose chat thread layouts
|
||||
│ ├── ConversationItem.kt # Adaptive cards (call, music, maps, thinking, attachments)
|
||||
│ ├── HandsFreeBar.kt # Animated floating orb with pulse glow rings
|
||||
│ ├── SettingsScreen.kt # Config UI with capability badges and download progress
|
||||
│ └── UserInputField.kt # Input box with file attachment preview
|
||||
└── theme/ # Colors, typography, dark/light themes
|
||||
│ ├── ChatScreen.kt # Main chat screen logic
|
||||
│ ├── ChatLayout.kt # Layout definitions for chat
|
||||
│ ├── ConversationItem.kt # Chat item UI elements
|
||||
│ ├── HandsFreeBar.kt # Voice UI overlay
|
||||
│ ├── SettingsScreen.kt # Settings layout and configuration
|
||||
│ └── UserInputField.kt # Text input and file picker UI
|
||||
└── theme/ # Theme, styling, and typography
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Setup
|
||||
## Setup
|
||||
|
||||
**Prerequisites**
|
||||
|
||||
- Android Studio
|
||||
- Android device or emulator running API 26+
|
||||
- Device or emulator (Android 8.0+ / API 26+)
|
||||
- Groq API key (or another OpenAI-compatible endpoint key)
|
||||
- YouTube Data API v3 key
|
||||
- Groq API key (or any OpenAI-compatible endpoint)
|
||||
|
||||
**Steps**
|
||||
|
||||
1. Clone the repository:
|
||||
1. Clone this repository:
|
||||
```bash
|
||||
git clone https://github.com/your-username/ai-assistant-android.git
|
||||
cd ai-assistant-android
|
||||
```
|
||||
|
||||
2. Open the project in Android Studio and allow Gradle to sync.
|
||||
|
||||
3. Add your API keys (see [API Configuration](#-api-configuration) below).
|
||||
|
||||
4. Connect a device or start an emulator, then click **Run**.
|
||||
|
||||
5. To use the app as your system assistant: go to **Settings → Apps → Default Apps → Digital assistant app** and select this app.
|
||||
2. Open the project in Android Studio and let files sync.
|
||||
3. Configure your API keys (see details in the API Configuration section).
|
||||
4. Run the app on your device.
|
||||
5. (Optional) Set the app as your default assistant under **Settings → Apps → Default Apps → Digital assistant app**.
|
||||
|
||||
---
|
||||
|
||||
## 🔑 API Configuration
|
||||
## API Configuration
|
||||
|
||||
Two methods are available — use whichever fits your workflow.
|
||||
|
||||
**Option A — `local.properties` (recommended for development)**
|
||||
|
||||
Create or edit `local.properties` in the project root:
|
||||
Choose one of these methods to configure your API keys:
|
||||
|
||||
**Option 1: Using `local.properties`**
|
||||
Add the keys to your `local.properties` file:
|
||||
```properties
|
||||
YOUTUBE_API_KEY=YOUR_YOUTUBE_API_KEY
|
||||
GROQ_API_KEY=YOUR_GROQ_API_KEY
|
||||
EDGE_TTS_SUBSCRIPTION_KEY=YOUR_EDGE_TTS_SUBSCRIPTION_KEY
|
||||
```
|
||||
Gradle will load these during the build. (Make sure this file is not committed to git).
|
||||
|
||||
Gradle reads these at build time and exposes them as `BuildConfig` fields. This file should already be in `.gitignore` — verify before committing.
|
||||
**Option 2: In-App Settings**
|
||||
Open the app, go to **Settings**, and paste your keys directly into the input fields. This does not require rebuilding the app.
|
||||
|
||||
**Option B — In-app Settings**
|
||||
|
||||
Install the app, open the **Navigation Drawer**, navigate to **Settings**, and paste your keys into the respective fields. No rebuild required.
|
||||
|
||||
**Custom LLM Endpoint**
|
||||
|
||||
In Settings, you can replace the default Groq endpoint with any OpenAI-compatible API URL (Ollama, LM Studio, Anthropic, Gemini, etc.). The app will probe the endpoint for vision support and available model lists automatically.
|
||||
**Customizing the LLM Endpoint**
|
||||
In the settings page, you can change the default URL to any OpenAI-compatible server (like Ollama, LM Studio, Anthropic, or Gemini). The app automatically checks if the custom endpoint supports images.
|
||||
|
||||
---
|
||||
|
||||
## 🔒 Permissions
|
||||
|
||||
The app requests permissions at runtime, only when a feature that needs them is first used.
|
||||
## Permissions
|
||||
|
||||
| Permission | Required For |
|
||||
|---|---|
|
||||
| `RECORD_AUDIO` | Voice input and VAD processing |
|
||||
| `READ_CONTACTS` | Fuzzy contact search for phone calls |
|
||||
| `CALL_PHONE` | Initiating calls |
|
||||
| `ACCESS_FINE_LOCATION` | Weather forecasts, navigation |
|
||||
| `MEDIA_PROJECTION` | Screen vision capture |
|
||||
| `BLUETOOTH` / `BLUETOOTH_CONNECT` | Bluetooth SCO headset routing |
|
||||
| `POST_NOTIFICATIONS` | Screen capture service notification |
|
||||
| `RECORD_AUDIO` | Listening to voice input |
|
||||
| `CAMERA` | Accessing the camera for live camera sharing |
|
||||
| `READ_CONTACTS` | Searching contact names to make calls |
|
||||
| `CALL_PHONE` | Making phone calls |
|
||||
| `ACCESS_FINE_LOCATION` | Checking weather and generating navigation routes |
|
||||
| `MEDIA_PROJECTION` | Recording the screen for screen vision |
|
||||
| `BLUETOOTH` / `BLUETOOTH_CONNECT` | Supporting Bluetooth headsets |
|
||||
| `POST_NOTIFICATIONS` | Showing the screen capture status bar notification |
|
||||
|
||||
---
|
||||
|
||||
## 📊 Supported Features Table
|
||||
## Supported Features Table
|
||||
|
||||
| Feature | Offline | Cloud | Native Android | AI / LLM |
|
||||
|---|:---:|:---:|:---:|:---:|
|
||||
| Voice recognition (VAD) | ✅ | | | |
|
||||
| Voice activity detection (VAD) | ✅ | | | |
|
||||
| Speech-to-text (Parakeet) | ✅ | | | |
|
||||
| Speech-to-text (Cloud) | | ✅ | | |
|
||||
| Intent classification | ✅ | | | |
|
||||
| Language translation | ✅ | | | |
|
||||
| Offline TTS (VITS) | ✅ | | | |
|
||||
| Edge TTS (neural) | | ✅ | | |
|
||||
| Conversation storage (encrypted) | ✅ | | | |
|
||||
| Neural TTS (Edge) | | ✅ | | |
|
||||
| Encrypted storage | ✅ | | | |
|
||||
| Phone calls | | | ✅ | |
|
||||
| Alarm & reminder scheduling | | | ✅ | |
|
||||
| Alarms & reminders | | | ✅ | |
|
||||
| Google Maps navigation | | | ✅ | |
|
||||
| Music playback (YouTube) | | ✅ | | |
|
||||
| YouTube music playback | | ✅ | | |
|
||||
| Weather forecasts | | ✅ | | |
|
||||
| Contact fuzzy matching | ✅ | | ✅ | |
|
||||
| Contact matching | ✅ | | ✅ | |
|
||||
| Screen vision | | | ✅ | ✅ |
|
||||
| General Q&A | | | | ✅ |
|
||||
| Custom LLM endpoint | | ✅ | | ✅ |
|
||||
| File & image attachments | | | | ✅ |
|
||||
| Live camera sharing | | | ✅ | ✅ |
|
||||
| General chat | | | | ✅ |
|
||||
| Custom LLM endpoints | | ✅ | | ✅ |
|
||||
| Image & file attachments | | | | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## 🛡️ Privacy & Security
|
||||
## Privacy & Security
|
||||
|
||||
All personally identifiable data stays on device or is explicitly sent by the user's chosen LLM endpoint.
|
||||
Your data stays on your device unless it is sent to your selected LLM provider.
|
||||
|
||||
**Database encryption**
|
||||
On first launch, the app generates a 256-bit AES key inside the hardware-backed Android KeyStore (TEE or Secure Element) under the alias `com.app.assistant.database_key`. Every message body and chat title is encrypted with AES-256-GCM before being written to the Room database. Each record gets a unique initialization vector — no two ciphertexts share a nonce.
|
||||
**Database Encryption**
|
||||
The app generates an AES-256 encryption key inside Android's secure hardware Keystore. All messages and chat titles are encrypted before saving. Each record uses a unique initialization vector.
|
||||
|
||||
**File encryption**
|
||||
Attached documents, images, and screen frames are encrypted with the same AES-GCM scheme before being written to the app's private storage directory. The unencrypted bytes are never persisted to disk.
|
||||
**File Encryption**
|
||||
All saved images, documents, and screenshots are encrypted before they are written to disk.
|
||||
|
||||
**Deletion**
|
||||
Deleting a message or chat group triggers a cascading cleanup: the repository decrypts the stored file paths, deletes the corresponding files from disk, and removes all database records. No orphaned ciphertext is left behind.
|
||||
**Secure Deletion**
|
||||
When you delete a message or chat, the app deletes the database records and removes the associated files from disk.
|
||||
|
||||
**Voice data**
|
||||
Raw PCM audio is never stored or transmitted. VAD evaluation happens entirely in memory on-device. Only the recognized text is passed downstream.
|
||||
**Voice Processing**
|
||||
Your raw voice audio is never stored or sent anywhere. Voice detection is handled entirely in memory on the device, and only the transcribed text is processed.
|
||||
|
||||
---
|
||||
|
||||
## 🗺️ Roadmap
|
||||
## Known Issues
|
||||
|
||||
- [ ] Wake-word activation ("Hey Assistant") using on-device detection
|
||||
- [ ] Multi-user encrypted profile support
|
||||
- [ ] Wear OS companion for hands-free wrist control
|
||||
- [ ] Plugin system for custom intent handlers
|
||||
- [ ] Per-conversation LLM model switching
|
||||
- [ ] Background conversation summarization
|
||||
- [ ] Accessibility service mode (no screen capture permission needed)
|
||||
- **Barge-In Interruption:** Sometimes the barge-in/interruption feature does not trigger correctly when speaking over the assistant. Work is underway to refine this.
|
||||
- **Translation Disabled:** The translation feature is currently disabled due to conflicts with streaming text output. A new approach is being developed.
|
||||
- **Settings Classification:** Settings-related commands (unlike Call, Play Song, etc., which work) do not have a separate action yet. Instead, they are classified under `OTHER` and sent directly to the LLM. A new approach is being worked on to support settings actions natively.
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
## Roadmap
|
||||
|
||||
Contributions are welcome. If you're fixing a bug, open a PR directly. For new features or significant changes, open an issue first to discuss the approach.
|
||||
|
||||
1. Fork the repository and create a feature branch.
|
||||
2. Follow the existing code style (Kotlin idioms, single-responsibility ViewModels).
|
||||
3. Open a pull request with a clear description of the change and why it was made.
|
||||
|
||||
Please ensure your changes do not introduce any plaintext storage of user data or bypass the encryption layer.
|
||||
- [ ] Android XR support for smart glasses
|
||||
- [ ] Wear OS support for wrist control
|
||||
- [ ] Category for sending text messages through actions
|
||||
- [ ] Automatic summary of older conversations
|
||||
- [ ] Screen capture using Accessibility Services to avoid permission prompts
|
||||
|
||||
---
|
||||
|
||||
## 📜 License
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome. Please open an issue to discuss new features before making changes, or open a pull request directly for bug fixes.
|
||||
|
||||
1. Fork the repository and create a branch.
|
||||
2. Follow the project's coding style (standard Kotlin, MVVM architecture).
|
||||
3. Submit a pull request explaining your changes.
|
||||
|
||||
Make sure your changes do not store unencrypted user data or bypass security features.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the [MIT License](LICENSE).
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 5.9 MiB After Width: | Height: | Size: 3.9 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 85 KiB After Width: | Height: | Size: 78 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 62 KiB After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 105 KiB After Width: | Height: | Size: 136 KiB |
Loading…
Add table
Add a link
Reference in a new issue