554
submitted 4 months ago* (last edited 4 months ago) by ForgottenFlux@lemmy.world to c/technology@lemmy.world

Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.

Last week, 404 Media noticed that search engines that weren't Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: "Reddit believes in an open Internet, but not the misuse of public content." Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.

The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).

In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It's unclear how much Reddit's OpenAI deal is worth.

Huffman said:

Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.

“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.

you are viewing a single comment's thread
view the rest of the comments
[-] catloaf@lemm.ee 41 points 4 months ago

And yet reddit is happy to make money off our content for free.

Or at least it did. Personally I overwrote and deleted all my content a while back.

[-] iAmTheTot@sh.itjust.works 15 points 4 months ago

You think that Reddit didn't already have the previous content saved?

[-] mp3@lemmy.ca 18 points 4 months ago

Bingo, the only winning move is not to play at all and stop using Reddit.

[-] dual_sport_dork@lemmy.world 9 points 4 months ago

Everyone always says this like it's some kind of gotcha, but all of my nuked posts still have my "fuck you, reddit" content and haven't been reverted. It's been nearly exactly a year.

Maybe reddit has an offline copy of my old content and that of others somewhere, but if so they'd be handing that directly over to whoever under some kind of agreement -- that certainly wouldn't be the subject of any kind of site crawling which is the crux of the issue here.

[-] Womble@lemmy.world 6 points 4 months ago

it never was deleted, all that happened is that an extra line was added to a database that said "comment 65432426542654 now should be displayed as "fuck you, reddit" rather than the original text". The original post is still in an earlier row available to reddit, it just isnt being displayed on their web page.

[-] stevedidwhat_infosec@infosec.pub 5 points 4 months ago

You’re ignoring the idea that they could still be working on a way to restore content and haven’t completed that process yet

Or that they could start feeding your archived (not cached) data directly to the AI companies anyway for a price

IMO, you can win by jamming your “transmissions” with noise. It’s easier to hide in noise as noise than it Is to be silent IMO. Muddy the waters as it were

[-] finley@lemm.ee 3 points 4 months ago

You’re ignoring the idea that they could still be working on a way to restore content and haven’t completed that process yet

there's no evidence to suggest this, though.

[-] stevedidwhat_infosec@infosec.pub 1 points 4 months ago

Content is absolutely archived and they have financial incentive to restore the quality of their “knowledge base”

That’s a fair amount of circumstance and motivation to support my idea, regardless of tangible evidence

[-] finley@lemm.ee 0 points 4 months ago

Motivation and circumstance, absent actual evidence, does not make for a convincing argument.

[-] stevedidwhat_infosec@infosec.pub 1 points 4 months ago

Alright well I guess evidence is needed before we can have ideas - crazy

[-] finley@lemm.ee 0 points 4 months ago

No, it just means that they are no more than ideas at this point

[-] stevedidwhat_infosec@infosec.pub 2 points 4 months ago* (last edited 4 months ago)

Right, which means it can be fairly considered when discussing the real crux of the issue with AI and big tech companies right now, which is the monetization of other peoples content.

If we’re discussing this, we should be looking at whether or not companies are doing this, given they have motive and specific, relevant circumstance to enact such behavior.

Lack of evidence means you need to investigate for said evidence. It does not mean you should not investigate. Privacy advocates, members of any org/cert with an ethics statement should be blowing the whistle on any kind of activity that would mean a users data is not being deleted upon their request, especially considering reddits global usage.

[-] finley@lemm.ee -1 points 4 months ago* (last edited 4 months ago)

By all means investigate, I’m just saying there has yet to be presented any actual evidence. I look forward to seeing whatever you may discover.

[-] iAmTheTot@sh.itjust.works 2 points 4 months ago

I certainly wasn't implying that they were going to revert your comments.

[-] finley@lemm.ee 3 points 4 months ago* (last edited 4 months ago)

i went looking for old comments and posts i had made after i overwrote then wiped them. They're still gone. i looked again several months later, and they were still gone.

so, unless reddit did a massive restore of everyone's comments/posts except for my 4 accounts, then i don't believe they did it at all except for a select number of top contributors who deleted their content.

[-] Mic_Check_One_Two@reddthat.com 6 points 4 months ago

On the other side of the same coin: When I mass edited my comments before quitting Reddit, I got site-banned. Basically, my first account’s automated edit got me auto-banned from several subs with pro-spez mods. Some subs had set their automod to detect when people were using the more popular methods of auto-editing, and set the automod to ban for using them. Then when I did the same with my second (and third, and fourth, and fifth, etc…) account, it almost immediately got site-banned for ban evasion.

Basically, account 1 was banned from a sub, so when account 2 started doing the same thing on the same IP address, it was flagged as ban evasion. And ban evasion is one of the few things that will get you banned site-wide instead of just from a specific sub.

I went back and checked a few months ago, and all of those site bans were lifted and the edits were undone. Likely because a site ban prevents the comments from showing up (which hurts Reddit’s bottom line, because they show up as a bunch of [removed] comments instead,) but also prevented any of the edits from actually being published. So when they lifted the site ban (to get those old comments to show back up again) it was as if I had never edited them at all. I had probably a million karma spread across my various accounts. I was extremely active at one point, so Reddit had a direct incentive to unban those accounts with literal thousands of comments.

[-] BorgDrone@lemmy.one 5 points 4 months ago

You are assuming edits overwrite existing content. Instead of overwriting, they could just store the edited post as a new entry in the database with a higher version number. Then, you only show the latest version of each post to the end users while keeping the older versions available die Reddit’s own use.

In fact, it is extremely likely they do this. It is basically a necessity if you want to be able to properly moderate a site like Reddit. Otherwise you could simply post spam or unsavory content, and then overwrite it with something benign an hour or so later, before there were enough reports and a moderator would have gotten a chance to review it.

[-] finley@lemm.ee -3 points 4 months ago

You are assuming edits overwrite existing content

i have seen no evidence to suggest otherwise, but thanks for sharing your theories

In fact, it is extremely likely they do this

based on what evidence? your baseless speculation?

[-] BorgDrone@lemmy.one 5 points 4 months ago

The fact that they managed to restore overwritten posts after people started to delete their history.

[-] finley@lemm.ee 0 points 4 months ago* (last edited 4 months ago)

this could also be explained by sketchy scripts failing to completely delete posts/comments, which i even noticed myself when checking that they had done their jobs properly. as i mentioned in another comment, i had to run the shredder scripts several times for complete overwrite/deletion. or it could be database errors failing to register edits/deletions due to extremely heavy loads at the time. it could be a lot of things.

the point is that we don't have any direct evidence of what it actually was, just a lot of circumstantial evidence and a lot of speculation. nothing definitive.

[-] BorgDrone@lemmy.one 7 points 4 months ago

Reddit used to be open source. There is still a copy of that source available on github. It’s 7 years old so it’s probably significantly different from what they are running now. Still, it gives some insight into the design.

For example, deleted comments aren’t deleted, it just sets a deleted flag. Example code that shows this.

I haven’t dug around the code enough to figure out how editing works, it’s Python code so an unreadable mess. The database design also seems very strange. It’s like they built a database system on top of a database.

[-] ChickenLadyLovesLife@lemmy.world 2 points 4 months ago

For example, deleted comments aren’t deleted, it just sets a deleted flag.

FWIW even when you properly delete something from a database table, the deleted row can be reconstructed from the audit tables. And even if that weren't the case, databases are regularly backed up to tape drives or whatever - when people delete or munge all their comments, Reddit doesn't go back into all the backups and make the same changes there. In fact, I would imagine that when they sell their shit to companies for AI training, they sell old pre-AI backups rather than a latest copy.

[-] finley@lemm.ee -2 points 4 months ago* (last edited 4 months ago)

This is not evidence that overwritten and deleted comments could be restored to the original state. Moreover, that points to the original source code of Reddit, not the current code of Reddit.

This is also not evidence that deleted or overwritten and deleted comments have been restored. This is merely evidence that, at one time, this is how deleted comments used to be handled.

All this is evidence of is, as you put it, things are very strange in the code.

[-] BorgDrone@lemmy.one 5 points 4 months ago

I never claimed it was evidence of how it currently works, only that it gives some insight into how Reddit was designed. I would be very surprised if they changed this aspect of the design. It makes sense to not delete comments or edits for reasons I mentioned before. Unfortunately we won’t know for sure unless Reddit confirms it.

[-] iAmTheTot@sh.itjust.works 5 points 4 months ago

As I said in another comment, I was not suggesting that Reddit would restore your comments to public view.

[-] finley@lemm.ee 1 points 4 months ago* (last edited 4 months ago)

there's no evidence to suggest that, either.

[-] SpaceNoodle@lemmy.world 4 points 4 months ago

Except for incidents like This

[-] magic_lobster_party@kbin.run 3 points 4 months ago

There’s no evidence of the contrary either.

[-] _haha_oh_wow_@sh.itjust.works 4 points 4 months ago

It seemed to happen to some people but I wouldn't be surprised it it was just some sort of coincidental database fuck-up

[-] finley@lemm.ee 2 points 4 months ago* (last edited 4 months ago)

i suspect that was more likely incomplete deletions than reddit restoring content. those scripts were pretty janky. i had to run mine several times to get everything, as it didn't work fully the first couple of times. same with the overwrites. took a few times for those to get everything, especially on older accounts with lots of posts and comments.

[-] _haha_oh_wow_@sh.itjust.works 3 points 4 months ago

I'd read some claims that posts appeared to be deleted but then later came back. Could've even been some sort of caching shenanigans with their local browser though I guess.

[-] finley@lemm.ee 1 points 4 months ago* (last edited 4 months ago)

oh, right. i suppose that's possible. i've seen similar browser cache fuckery on other sites before.

[-] finley@lemm.ee 0 points 4 months ago* (last edited 4 months ago)

as i said in a previous comment:

so, unless reddit did a massive restore of everyone’s comments/posts except for my 4 accounts, then i don’t believe they did it at all except for a select number of top contributors who deleted their content.

but there's no evidence they're keeping everyone's deleted-but-restored comments from public view or whatever it is you're suggesting. or even anything past whaat this one person found. in fact, there isn't even any evidence that what happened to this user was intentional and not a bug or some other fluke.

sure, reddit would have a vested interest in doing this, and what you've presented is suspicious, but it's hardly conclusive of anything. all it does is raise more questions. but it doesn't provide answers.

[-] Animoscity@lemmy.world 1 points 4 months ago

Yep that’s how it works. Older content past a certain date is cached which is why you can’t comment or post on some old posts.

[-] dmtalon@infosec.pub 3 points 4 months ago

They aren't making money off my content anymore/in the future

this post was submitted on 01 Aug 2024
554 points (97.8% liked)

Technology

60123 readers
2671 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 2 years ago
MODERATORS