User-agent: * Disallow: /chatbot/ Allow: / Crawl-delay:3 Sitemap:/sitemap_index.xml Disallow: /wp-admin/ Disallow: /wp-content/plugins/ Disallow: /wp-includes/js/* Disallow: /author/* Disallow: /comment-page-* Disallow: *.pdf