Archon creates almost 8000 code examples 😳

[Chillbruhhh]

Came here to say im quite impressed with how well Archon handled this task, almost 8000 code examples were crawled and currently chunking, took all day (my computer goes to sleep after two hours so i had to keep coming back to it) but it did it!

This was on the Twilio documentation. What's the biggest thing you've crawled and chunked?

[TimHuey]

What level did you crawl to? I tried to crawl Archon's docs and selected level 3. Seems excessive, but if I'm understanding the recursive nature the search, it searches each link, then searches each link in that link, then searches each link in that link for a total of a depth of 3.

so for Archon's first link ex:
https://github.com/coleam00/Archon ->
https://github.com/coleam00/Archon/tree/main/.claude ->
https://github.com/coleam00/Archon/tree/main/.claude/agents ->
https://github.com/coleam00/Archon/blob/main/.claude/agents/codebase-analyst.md

So in order to actually get the codebase-analyst.md into your supabase you would need a depth of 3 or possibly 4 depending on where it starts the depth count.

Maybe my understanding is incorrect, but just to get that level of webcrawling it took a huge amount of time. Depth of 3 on Archon came up with 3777 pages.

[Chillbruhhh]

i crawled twilios llms.txt on level 2 it crawls all the links there on the page https://www.twilio.com/docs/llms.txt