Website Crawling for Knowledge

Your AI agents can now learn from any website. Enter a URL, the system discovers all pages, and you pick which ones to import into your knowledge base.

How it works

1. Enter a website URL in the Knowledge Base
2. The crawler discovers all accessible pages and shows them in an interactive tree
3. Select the pages you want — product pages, FAQ sections, pricing, documentation
4. Selected pages are processed and made available to your AI agents

When a customer asks a question, the agent searches the imported website content alongside your other knowledge sources and responds with accurate, up-to-date information.

Use cases

• Your own website — your AI agent should know everything on your website. Import your product pages, service descriptions, and FAQ
• Competitor sites — import competitor pricing or feature pages so your agent can answer comparison questions
• Documentation — point it at your help centre or documentation site
• Partner sites — import relevant pages from partner or supplier websites

Content preview

Before importing, preview the extracted content from any page to verify it looks right. The system handles navigation menus, footers, and other noise — extracting just the meaningful content.

Part of Knowledge Base

Website sources sit alongside uploaded files and Google Drive imports in your Knowledge Base. All three sources feed into Knowledge Skills that you assign to agents and channels. One unified system for everything your AI needs to know.

The Documents section has been renamed to Knowledge Base to better reflect what it does.