back to changelog

ChatBotKit Now Supports Crawling Sitemap.xml and Spidering Full Websites

Learn about the latest update to ChatBotKit, which now supports crawling sitemap.xml and spidering full websites. With this new feature, our chatbots can access even more information and provide even more context to users.

We are excited to announce that ChatBotKit now supports both crawling sitemap.xml as well as spidering the full website. With this new feature, websites that do not have sitemaps can now be used in datasets.

As you may already know, the dataset is the knowledge-base used by advanced conversational AI chatbots to provide additional context. And now, with our latest update, our chatbots can crawl and spider any website, regardless of whether or not they have sitemaps.

This means that our chatbots will be able to access even more information and provide even more context to users. We believe that this update will significantly improve the overall performance of our chatbots.