Website Import & Crawling
ChatBotKit can turn any website into knowledge for your AI. Crawl a site's sitemap.xml to import its listed pages, or spider the full website when no sitemap exists. The importer also handles modern sites that rely heavily on JavaScript for dynamic content, so pages that render client-side are captured too.
Imported pages become dataset records - the knowledge base your bots draw on during conversations - so your AI can answer grounded in the content of your website.
What You Can Do
- Crawl a sitemap: Import the pages listed in a site's
sitemap.xml. - Spider full sites: Capture websites that have no sitemap by following their links.
- Handle JavaScript: Import dynamic, client-rendered pages, not only static HTML.
- Build a knowledge base: Imported content becomes dataset records for retrieval during conversations.
How It Works
You point the importer at a website, and ChatBotKit crawls it - either through the sitemap or by spidering links across the site. Page content is extracted and stored as dataset records, which your bots retrieve through retrieval-augmented generation when answering questions. For JavaScript-heavy sites, the importer renders pages so dynamic content is included rather than missed.
Setup
Create a dataset and use the website importer to add a site. Provide the URL, choose sitemap crawling or full-site spidering, and let ChatBotKit import the pages. The resulting records are immediately available to any bot that uses the dataset.
Practical Uses
Website import is ideal for building support and sales bots from your existing documentation, help center, product pages, or marketing site. Because the importer handles sitemaps, full-site spidering, and JavaScript rendering, you can bring almost any web content into your AI's knowledge base with minimal setup.