Is there a way to crawl a site without specifying an selectors and schema?

[Tony20221]

Is there a way to crawl a site without specifying an selectors and schema? The plan is to use the app with any site and without knowing what the site is when the app is running. Start with a home page, parse the links and get the pages for those links and so on.

[Marcel0024]

Probably if you send the page to an LLM. Not happening with this library

[Tony20221]

Probably if you send the page to an LLM. Not happening with this library

What do you mean? I am trying find a library that can crawl a website. I don't care about the scraping functionality. I am doing that with my code. I want to get the full page for every page.