Obsidian RAG Starter

Transform your Obsidian vault into a queryable knowledge base in minutes 🚀

💡 What Is This?

In Simple Terms: This tool makes your Obsidian notes searchable with AI. Instead of manually searching through hundreds of notes, you can ask questions like "What did I write about productivity?" and get intelligent answers based on YOUR notes.

Example:

You ask: "What are my thoughts on morning routines?"

AI responds: "Based on your notes, you prefer starting with meditation at 6 AM
(mentioned in 'Daily Habits.md'), followed by journaling. In 'Productivity
Systems.md' you noted that morning exercise gives you more energy..."

It's like having ChatGPT, but it only knows about YOUR notes.

✨ Features

🔍 Intelligent Search

• Semantic search - Find content by meaning, not just keywords
• Natural language queries - Ask questions like "What did I learn about React patterns?"
• Contextual results - Get relevant chunks with source citations

📚 Obsidian Native

• Wikilink aware - Understands [[internal links]] and preserves connections
• Tag integration - Searches through your #tags seamlessly
• Heading hierarchy - Maintains document structure and context
• Frontmatter support - Extracts and uses metadata from your notes

🛠️ Developer Friendly

• TypeScript first - Fully typed for better DX and fewer bugs
• Pluggable architecture - Easy to add new embedding providers or vector stores
• Docker Compose - One-command local development environment
• API-first design - Use via CLI, web UI, or integrate with your own tools

🏗️ Production Ready

• Multiple storage options - Local pgvector or hosted Supabase
• Batch processing - Efficiently handle large vaults
• Error handling - Graceful failures with detailed logging
• Rate limiting - Built-in API protection

🚀 Quick Start

Choose your setup path based on your needs:

🌟 Option A: Hosted Setup (Recommended for Beginners)

Uses Supabase for zero-configuration vector storage

Prerequisites

• Node.js 18+
• OpenAI API key
• Supabase account (free tier available)

📋 Step-by-Step Setup Guide

Step 1: Download the Project

git clone https://github.com/yourusername/obsidian-rag-starter.git
cd obsidian-rag-starter
npm install

What this does: Downloads the code and installs required packages (takes ~1 minute)

---

Step 2: Get Your API Keys

You need 2 things:

2A. OpenAI API Key (for AI responses and embeddings)

1. Go to https://platform.openai.com/api-keys
2. Sign up or log in
3. Click "Create new secret key"
4. Copy the key (starts with sk-...)
5. Cost: About $0.01 per 100 queries (very cheap!)

2B. Supabase (Free Database)

1. Go to https://supabase.com
2. Click "Start your project" (free, no credit card)
3. Create a new project:
   • Name it something like "obsidian-rag"
   • IMPORTANT: Save the database password!
   • Pick the region closest to you
   • Wait 2 minutes for setup
4. Once ready, go to Settings → API in left sidebar
5. Copy these TWO things:
   • Project URL: https://xxxxx.supabase.co
   • service_role key: The long string under "service_role" (starts with eyJ...)

---

Step 3: Save Your Keys

# Create your environment file
cp .env.example .env

Now open .env in any text editor (VS Code, Notepad, TextEdit, etc.) and fill in:

OPENAI_API_KEY=sk-proj-xxxxxxxxxxxxx
SUPABASE_URL=https://xxxxx.supabase.co
SUPABASE_SERVICE_ROLE_KEY=eyJhbGc...xxxxx

Note: Don't use quotes around the values!

---

Step 4: Create Configuration

npx obsidian-rag init

This creates obsidian-rag.yml. Open it and change ONLY this line:

vault:
  path: /Users/YourName/Documents/ObsidianVault # ← PUT YOUR VAULT PATH HERE - you can find this by going to your Obsidian vault in your Finder/Folders of your Mac/Windows. You can also find the path by right clicking on the folder you want to index in your Obsidian app and clicking "Reveal in Finder".

How to find your vault path:

• Mac: Right-click vault folder → "Get Info" → copy where it says "Where:"
• Windows: Right-click vault folder → "Properties" → copy "Location:"
• In Obsidian: Settings → Files and Links → look for vault location

---

Step 5: Set Up Database Tables

npx obsidian-rag setup

This will show a link like https://supabase.com/dashboard/project/xxxxx

1. Click that link (opens your Supabase dashboard)
2. Click SQL Editor (left sidebar)
3. Click New query button
4. Go back to your code folder and open scripts/supabase-schema.sql
5. Copy ALL the content from that file
6. Paste it in the SQL Editor
7. Click RUN button (green, bottom-right)
8. You should see "Success. No rows returned" ✅

---

Step 6: Index Your Notes

npx obsidian-rag index

This reads all your Obsidian notes and makes them searchable. You'll see:

• "Found X markdown files"
• Progress updates as it processes
• "✅ Indexing completed successfully!"

First time: Takes 1-5 minutes depending on vault size
Cost: About $0.001 per 100 notes

---

Step 7: Start Using It!

Option A - Web Interface (Recommended):

npm run ui

Then open http://localhost:3000 in your browser

Option B - Command Line:

npx obsidian-rag query "What are my thoughts on productivity?"

---

🚨 Troubleshooting

Problem	Solution
"Configuration file not found"	Run npx obsidian-rag init first
"Invalid API key"	Check .env file - no quotes around keys!
"Vault path does not exist"	Use full path in obsidian-rag.yml, not ~ or relative paths
"Connection refused" on web	Make sure you ran npm run ui first
Files not showing in VS Code	Click refresh icon in file explorer
"No results found"	Lower threshold: add --threshold 0.2 to query

🔧 Option B: Self-Hosted Setup (For Privacy & Control)

Uses local PostgreSQL with pgvector for complete data control

Prerequisites

• Node.js 18+
• Docker and Docker Compose
• OpenAI API key (or use mock provider for testing)

Setup Steps

# 1. Clone and install
git clone https://github.com/yourusername/obsidian-rag-starter.git
cd obsidian-rag-starter
npm install

# 2. Start local PostgreSQL with pgvector
docker-compose up -d
# This creates a local database with vector search capabilities

# 3. Configure environment
cp .env.example .env
# Edit .env and add:
# OPENAI_API_KEY=your_openai_key_here
# DATABASE_URL=postgresql://postgres:password@localhost:5432/obsidian_rag

# 4. Create config file and customize
npx obsidian-rag init
# Edit obsidian-rag.yml:
# - Set vault.path to your Obsidian vault location
# - Change vectorStore.type to "pgvector"
# - Update vectorStore.connectionString to match DATABASE_URL

# 5. Index your vault and start querying!
npx obsidian-rag index
npx obsidian-rag query "What are my thoughts on productivity?"

# 6. Start the web interface
npm run ui
# Open http://localhost:3000

🧪 Option C: Demo Mode (No API Keys Needed)

Perfect for trying out the system with mock data

# 1. Clone and install
git clone https://github.com/yourusername/obsidian-rag-starter.git
cd obsidian-rag-starter
npm install

# 2. Use mock embedding provider (no API key needed)
npx obsidian-rag init
# Edit obsidian-rag.yml and set:
# embedding.provider: "mock"

# 3. Try it with the sample vault
npx obsidian-rag index examples/sample-vault
npx obsidian-rag query "What is Obsidian RAG?"
npx obsidian-rag serve

📖 Usage Guide

Command Line Interface

# Index your vault
obsidian-rag index [vault-path] [options]
  --force                Re-index all files
  --dry-run              Show what would be indexed
  --cost-estimate        Estimate embedding costs

# Query your knowledge base
obsidian-rag query "your question" [options]
  --limit <number>       Max results (default: 5)
  --threshold <number>   Min similarity (default: 0.3)
  --items                Show individual chunks (old behavior)
  --sources-only         Just show sources
  --json                 JSON output

# Start web interface with search UI
npm run ui                # Starts both backend (port 3001) and frontend (port 3000)
# Open http://localhost:3000

# Or start API server only (for programmatic access)
obsidian-rag serve [options]
  --port <number>        Server port (default: 3001)
  --host <string>        Bind host (default: localhost)

# Manage configuration
obsidian-rag config validate    # Check config
obsidian-rag config show        # Display current config
obsidian-rag status             # Show system status

Web Interface (Recommended)

For most users: Use the web interface for a better experience than CLI:

npm run ui  # Starts everything you need

Then open http://localhost:3000 in your browser.

The web UI provides:

• 🤖 AI-generated answers (not just raw chunks)
• 📄 Source citations showing where info came from
• 🎯 Real-time search with similarity scores
• 📱 Mobile-friendly responsive design

Note: The serve command alone is for API-only access (advanced users). Most users should use npm run ui for the full web experience.

API Integration

The REST API makes it easy to integrate with other tools:

# Query endpoint
curl -X POST http://localhost:3001/api/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What are my project ideas?", "limit": 5}'

# Health check
curl http://localhost:3001/api/health

# Get statistics
curl http://localhost:3001/api/stats

⚙️ Configuration

Embedding Providers

OpenAI (Recommended)

embedding:
  provider: "openai"
  model: "text-embedding-3-small" # Cost-effective, high quality
  apiKey: "${OPENAI_API_KEY}"
  batchSize: 32

Cohere

embedding:
  provider: "cohere"
  model: "embed-english-v3.0"
  apiKey: "${COHERE_API_KEY}"
  batchSize: 96

Mock (for testing)

embedding:
  provider: "mock" # No API key needed

Vector Stores

Supabase (Recommended for beginners)

vectorStore:
  type: "supabase"
  connectionString: "${SUPABASE_URL}"
  # Requires SUPABASE_SERVICE_ROLE_KEY in environment

Local pgvector

vectorStore:
  type: "pgvector"
  connectionString: "postgresql://postgres:password@localhost:5432/obsidian_rag"

Advanced Configuration

See example-config.yml for all available options:

• Chunking strategies - Token limits, overlap, heading respect
• File filtering - Include/exclude patterns
• Server settings - CORS, rate limiting, ports
• Performance tuning - Batch sizes, connection pooling

🖥️ CLI Commands

setup - Database Setup

npx obsidian-rag setup

Tests connection and prepares database setup. For Supabase: Provides manual setup instructions pointing to scripts/supabase-schema.sql (one-time). For local pgvector: Fully automated table creation.

init - Configuration Setup

npx obsidian-rag init

Creates a configuration file with sensible defaults. Edit the generated obsidian-rag.yml to point to your vault location.

index - Vault Indexing

npx obsidian-rag index [options]
npx obsidian-rag index --dry-run        # Preview what will be indexed
npx obsidian-rag index --cost-estimate  # Estimate embedding costs
npx obsidian-rag index --force          # Rebuild entire index

query - Knowledge Base Query

npx obsidian-rag query "your question here" [options]
npx obsidian-rag query "What are my meeting notes?" --limit 10

serve - Web Interface

npx obsidian-rag serve [options]
npx obsidian-rag serve --port 3002  # Custom port

📁 Project Structure

obsidian-rag-starter/
├── src/                    # Core TypeScript source
│   ├── lib/               # Shared libraries
│   │   ├── types.ts       # TypeScript interfaces
│   │   ├── config.ts      # Configuration management
│   │   ├── parser.ts      # Markdown parsing & chunking
│   │   ├── embeddings.ts  # Embedding providers
│   │   ├── vector-db.ts   # Vector store abstractions
│   │   └── utils.ts       # Shared utilities
│   ├── commands/          # CLI command implementations
│   │   ├── index.ts       # Vault indexing
│   │   ├── query.ts       # Knowledge base querying
│   │   └── serve.ts       # API server
│   └── cli.ts            # Main CLI entry point
├── web/                   # Next.js web interface
│   ├── app/              # Next.js 14 app directory
│   │   ├── page.tsx      # Main search interface
│   │   └── layout.tsx    # App layout
│   └── package.json      # Web dependencies
├── examples/
│   └── sample-vault/     # Example Obsidian vault
├── scripts/
│   └── init-db.sql       # Database initialization
├── docker-compose.yml    # Local development setup
└── example-config.yml    # Configuration template

🤝 Contributing

We welcome contributions! Here are some ways to help:

🆕 Good First Issues

• Add Qdrant vector store support
• Implement Cohere embedding provider tests
• Add support for PDF files in vaults
• Create Docker image for easy deployment
• Add more chunking strategies
• Implement streaming responses in web UI
• Add authentication to web interface
• Create VS Code extension integration

🚀 Advanced Features

• Graph RAG with wikilink relationships
• Multi-modal support (images, diagrams)
• Collaborative team knowledge bases
• Integration with Obsidian plugins
• Advanced query planning and routing

Development Setup

# Clone the repo
git clone https://github.com/yourusername/obsidian-rag-starter.git
cd obsidian-rag-starter

# Install dependencies
npm install
cd web && npm install && cd ..

# Start local database (if using pgvector)
docker-compose up -d

# Build TypeScript
npm run build

# Run tests
npm test

# Start development
npm run dev

📊 Performance & Costs

Embedding Costs (approximate)

• 1,000 notes (~500k tokens): $0.01 with text-embedding-3-small
• 10,000 notes (~5M tokens): $0.10 with text-embedding-3-small
• 100,000 notes (~50M tokens): $1.00 with text-embedding-3-small

Performance Benchmarks

• Indexing: ~1,000 notes per minute (depends on API limits)
• Query time: <200ms typical response time
• Memory usage: ~100MB for CLI, ~200MB for web server
• Storage: ~1KB per chunk in vector database

🔒 Privacy & Security

• Local-first: Your notes never leave your machine unless you choose cloud providers
• API keys: Stored securely in environment variables, never in code
• Rate limiting: Built-in protection against API abuse
• CORS: Configurable origin restrictions for web interface
• No telemetry: We don't collect any usage data

🗺️ Roadmap

v0.2.0 - Enhanced Search

• Hybrid search (semantic + keyword)
• Query expansion and rephrasing
• Result re-ranking and filtering
• Search result explanations

v0.3.0 - Advanced Features

• Graph-based retrieval using wikilinks
• Multi-hop reasoning across notes
• Conversation memory and context
• Custom embedding fine-tuning

v1.0.0 - Production Ready

• Managed cloud hosting option
• Team collaboration features
• Enterprise security features
• Comprehensive documentation

❓ FAQ

Q: Does this work with any note-taking app? A: Currently optimized for Obsidian, but it works with any markdown files. Wikilinks and tags are Obsidian-specific features.

Q: Can I use this without OpenAI? A: Yes! You can use Cohere, or the mock provider for testing. We're working on local embedding model support.

Q: How much does it cost to run? A: For most personal vaults: <$1/month for embeddings. Vector storage is free with pgvector or ~$5/month with Supabase.

Q: Is my data secure? A: Yes! With local pgvector, everything stays on your machine. With Supabase, data is encrypted at rest.

Q: Can I deploy this for my team? A: Absolutely! Use Docker for easy deployment, or deploy the Next.js app to Vercel/Netlify.

🔗 Related Projects

• Obsidian - The knowledge management app this works with
• pgvector - PostgreSQL vector extension
• Supabase - Open source Firebase alternative
• LangChain - Framework for LLM applications
• Pinecone - Managed vector database alternative

📄 License

MIT License - see LICENSE file for details.

🙏 Acknowledgments

• The Obsidian team for creating an amazing knowledge management tool
• The pgvector team for the excellent PostgreSQL extension
• OpenAI for providing accessible embedding APIs
• The open source community for inspiration and feedback

---

Built with ❤️ for knowledge workers, researchers, and anyone who wants to unlock the full potential of their notes.

⭐ Star this repo if you find it useful!