🤖 CrawlBot - Transform Any Website into an Intelligent Knowledge Assistant - n8n workflow

Transform any website content into intelligent, queryable knowledge with this powerful n8n workflow.

CrawlBot automatically crawls all URLs from XML sitemaps, processes web content, converts them into vector embeddings for advanced retrieval augmented generation (RAG) applications, and enables intelligent chat-based querying of your website data.

🚀 Key Features

• Website Crawling: Automatically crawls all URLs found in XML sitemaps using the powerful Crawl4AI service
  

• Smart Content Extraction: Removes HTML tags and extracts clean, meaningful text content using AI-powered processing

• Vector Embeddings: Converts cleaned content into high-dimensional vector embeddings for semantic search capabilities

• Intelligent RAG System: Implements an agentic RAG system that searches through crawled content to provide accurate, context-aware responses

• Conversation Memory: Maintains chat history across sessions using PostgreSQL for contextual conversations

💡 Use Cases

• Documentation Sites: Transform technical documentation into an AI-powered knowledge assistant that can answer complex questions about your products or services
• Company Websites: Create intelligent chatbots that can answer questions about your company, products, and services based on your website content
• Knowledge Bases: Convert help centers and FAQ sections into conversational AI assistants for better customer support
• Content Archives: Make large content libraries searchable through natural language queries with semantic understanding
• Research Websites: Enable researchers to query academic content and publications through conversational interfaces
• News and Media Sites: Create AI assistants that can answer questions about archived articles and current content

⚙️ Requirements

• n8n instance (cloud or self-hosted)
• Crawl4AI service(free) running on locally
• OpenAI API key for embeddings and chat models
• Google Gemini API key for content cleaning
• Supabase account(free)

🔧 Easy Setup

The workflow comes with comprehensive documentation that guides you through:

• Setting up n8n (both local and cloud deployment options)
• Installing and configuring Crawl4AI service
• Configuring required credentials (OpenAI, Google Gemini, PostgreSQL/Supabase)
• Setting up the PostgreSQL database with pgvector extension
• Creating necessary tables and functions for vector storage
• Testing the crawling and chat functionality

💬 FAQ

Q: How does CrawlBot handle large websites?

A: CrawlBot processes URLs sequentially from the sitemap, with built-in retry logic and status monitoring to ensure reliable crawling of all pages.

Q: Can it crawl dynamic JavaScript websites?

A: Yes! Crawl4AI is designed to handle modern JavaScript-rendered websites and extract content after the page fully loads.

Q: How often is the content updated?

A: You can manually trigger the crawling process whenever needed. For automatic updates, you can schedule the workflow to run periodically.

Q: Can I limit crawling to specific sections of a website?

A: Yes! You can provide a sitemap that only includes the URLs you want to crawl, or modify the workflow to filter URLs based on patterns.

Q: How does the chat interface work?

A: The AI agent uses semantic search to find relevant content from the vector database and provides responses based on the crawled website data.

📞 Support

Need assistance? Connect with me on Twitter/X @victor_explore and send a DM for:

• Personalized setup assistance with this workflow
• Custom workflow development for your specific analytical needs

Transform website content into intelligent, queryable knowledge with CrawlBot today!