Learn how to use regular expressions (regex) to create effective search filters for web scraping in LearnWise.

Understanding regex and search filters is crucial for efficient web scraping in LearnWise. These tools allow you to precisely control which content is imported into your AI assistant's knowledge base. By mastering regex, you can:

1. Ensure your AI only learns from relevant pages, improving its accuracy and performance
2. Exclude outdated or irrelevant content, maintaining the quality of your assistant's knowledge
3. Navigate complex website structures to capture exactly the information you need
4. Save time and resources by automating the content selection process
5. Customize your AI's knowledge base to better serve your specific use case or industry

This guide explains the basics of regex and how to apply it to positive and negative search filters, empowering you to create a more focused and effective AI assistan

Step By Step

1. Understand the basics of regex
   • Learn common regex patterns like ^ (start of string), $ (end of string), .* (any characters), and \\d (digits)
2. Create positive search filters
   • Use regex to define which URLs to include in the scraping process
   • Example: ^https://www\\.example\\.com/product/.*$ to include all product pages
3. Create negative search filters
   • Use regex to define which URLs to exclude from the scraping process
   • Example: ^https://www\\.example\\.com/blog/2023/.*$ to exclude blog posts from 2023
4. Test your regex patterns
   • Use online regex testers to verify your patterns work as intended
   • Refine your filters as needed for more precise results
5. Apply filters in LearnWise

   • Enter your positive and negative search filters in the appropriate fields by clicking on ‘Add Website” and then expanding the “Advanced Settings” accordion.

     

     Screenshot of the “Add Website” modal within the LearnWise admin panel.

By mastering regex for search filters, you can efficiently control which content is imported into your LearnWise knowledge base, ensuring your AI assistant has access to the most relevant information.

<aside> 💡 If you have additional questions please reach out to [email protected]

</aside>