LLM Web Craler

John Doe

JD

Dashboard New Crawl Results Settings Chat Agent

Dashboard New Crawl Results Settings Chat Agent

Web Crawler

Website URL or Sitemap URL

Preferred Data Format

Max Links to Crawl

Crawl Options

Respect robots.txt

Crawl subpages

Filter out unnecessary information using LLM