79
submitted 1 year ago by 111@zerobytes.monster to c/foss@beehaw.org
you are viewing a single comment's thread
view the rest of the comments
[-] Destragras@kbin.social 12 points 1 year ago* (last edited 1 year ago)

I tried finding information on what indexer they are using. Are they using their own?

Edit: says this in the readme:

The commoncrawl organization for crawling the web and making the dataset readily available. Even though we have our own crawler now, commoncrawl has been a huge help in the early stages of development.

this post was submitted on 27 Aug 2023
79 points (100.0% liked)

Free and Open Source Software

18020 readers
113 users here now

If it's free and open source and it's also software, it can be discussed here. Subcommunity of Technology.


This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 2 years ago
MODERATORS