🤖 Open-source AI automation for everyone, powered by local LLMs
Installation • Documentation • Examples • Contributing • Community
AI Model Configuration and Theme Settings
Task History Overview
New Automation Task Creation Interface
DesktopAutoPilotX is your computer's new co-pilot – a free, open-source AI automation system that sees, understands, and interacts with your desktop just like you do. No cloud services, no API keys, just pure local AI power.
- 🧠 True AI Vision: Powered by LLaVA and Llama 2 for intelligent screen understanding
- 🎯 Context-Aware: Finds and interacts with UI elements naturally, no pixel coordinates needed
- 📊 Transparent: Real-time visualization and detailed logging of all actions
- 🛡️ Secure: Runs 100% locally - your data never leaves your machine
- 🔄 Self-Learning: Gets smarter with each task it handles
- Python 3.11+
- PostgreSQL
- Ollama with LLaVA and Llama 2 models
- Git
- Poetry (Python package manager)
# Clone the repo
git clone https://github.com/0xSG/DesktopAutoPilotX.git
cd DesktopAutoPilotX
# Install dependencies using Poetry
poetry install
# Activate the virtual environment
poetry shell
# Set up environment variables
cp .env.example .env
# Edit .env with your settings
# Initialize database
flask db upgrade
# Launch
python main.py# Install development dependencies
poetry install --with dev,test,docs
# Install pre-commit hooks
poetry run pre-commit install
# Run tests
poetry run pytest
# Run type checking
poetry run mypy .
# Format code
poetry run black .
poetry run ruff .# Build and run using Docker Compose
docker compose up -dDesktopAutoPilotX/
├── src/
│ └── desktop_autopilot/
│ ├── core/ # Core automation engine
│ ├── models/ # AI model integrations
│ ├── vision/ # Screen analysis components
│ └── utils/ # Utility functions
├── tests/ # Test suite
├── docs/ # Documentation
├── examples/ # Example scripts
├── docker/ # Docker configurations
├── pyproject.toml # Project dependencies and config
└── README.md # You are here
from desktop_autopilot import AutoPilot
pilot = AutoPilot()
# Example 1: Web Automation
@pilot.task
async def search_github():
"""Search for a repository on GitHub."""
await pilot.browser.open("https://github.com")
await pilot.keyboard.type("DesktopAutoPilotX")
await pilot.keyboard.press("Enter")
# Example 2: Desktop App Automation
@pilot.task
async def process_images():
"""Batch process images in a folder."""
await pilot.app.launch("PhotoEditor")
await pilot.folder.select("~/Pictures/batch")
await pilot.vision.wait_for("Process All")
await pilot.mouse.click()More examples in our cookbook 📚
Configure your AI providers in config/providers.toml:
[ollama]
base_url = "http://localhost:11434"
models = ["llava", "llama2"]
timeout = 30
[local]
model_path = "/path/to/models"
device = "cuda"
[api]
base_url = "https://api.example.com/v1"
timeout = 10# Required
DATABASE_URL=postgresql://user:pass@localhost:5432/dbname
FLASK_SECRET_KEY=your-secret-key
# Optional
LOG_LEVEL=INFO
OLLAMA_BASE_URL=http://localhost:11434
We believe great software is built together! Check out our Contributing Guidelines to get started.
- Fork the repository
- Create a new branch:
git checkout -b feature-name - Make your changes
- Run tests:
poetry run pytest - Format code:
poetry run black . && poetry run ruff . - Commit:
git commit -m "feat: add new feature" - Push:
git push origin feature-name - Open a Pull Request
DesktopAutoPilotX is MIT licensed, as found in the LICENSE file.
Made with ❤️ by the DesktopAutoPilotX Community