167

Media upload filter updates again (lemmy.world)

submitted 5 days ago by lwadmin@lemmy.world to c/lemmyworld@lemmy.world

16 comments fedilink hide all child comments

Hello World,

today, @db0@lemmy.dbzer0.com has provided an update to the media upload scanner we're using. This should reduce the amount of false positives blocked from being uploaded. We have deployed the updated version now.

While we do not have stats about false positives from before we implemented the scan when uploading, those changes did not change the overall data availability for us. Flagged images were still deleted, they were just still served by our cache in many cases. By moving this to the upload process, it has become much more effective, as previously images could persist in Cloudflare's cache for extended periods of time, while now they won't get cached in the first place.

Over the last week, we've seen a rate of roughly 6.7% uploads rejected out of around 3,000 total uploads. We'll be able to compare numbers in a week to confirm that this has indeed improved the false positive rate.

top 16 comments

sorted by: hot top controversial new old

[-] cm0002@lemmy.world 67 points 5 days ago

Lol my meme worked! :D

My unaltered meme now passes!

[-] LostXOR@fedia.io 31 points 5 days ago

Finally, the oppressive censorship of our right to free speech has been lifted! Parenthood!

[-] bassomitron@lemmy.world 9 points 5 days ago

I don't get it. Did the old filter used to censor images containing the word parenthood or something?

[-] cm0002@lemmy.world 5 points 5 days ago

Yea, I know of it doing it on the words parenthood, children and kids because I also had an earlier meme about before and after kids and all three of those would fail without censoring

[-] nroth@lemmy.world 8 points 5 days ago

6.7% is still very high

[-] m_f@midwest.social 13 points 5 days ago

Any details about the system? Is it something off-the-shelf, or custom-made? Is it using something fancy like transformer models? I'm guessing that would be too expensive

[-] MrKaplan@lemmy.world 13 points 5 days ago

https://lemmy.world/post/5397202

[-] m_f@midwest.social 2 points 5 days ago

Thanks!

[-] IMALlama@lemmy.world 3 points 5 days ago

I still can't upload unedited pictures from my pixel 3a via Jerboa :( For whatever reason, uploading them as-is results In a 403 response. Editing them in google photos (minor cropping), saving, and trying again would often result in success.

[-] cm0002@lemmy.world 9 points 5 days ago

Have you tried another Lemmy client? For scanner rejected media boost would report an error 400, Unknown error, not 403

You can double check by going to lemmy.world , click create post and then skip to upload an image (you don't need to fill anything out) and if you get a red popup with "External Validation Failed" that's when it got rejected

[-] IMALlama@lemmy.world 1 points 5 days ago* (last edited 5 days ago)

I tried using the lemmy.world web UI and got the following error: json.parse unexpected character at line 1 column 1 of the JSON file.

[-] IMALlama@lemmy.world 1 points 5 days ago

I also get an error using the web UI - json.parse unexpected character at line 1 column 1 of the JSON file.

[-] MrKaplan@lemmy.world 4 points 5 days ago

a JSON parsing error is certainly unexpected. I wonder if you happen to be triggering some automatic Cloudflare WAF rule.

could you try uploading the same file from a computer? before uploading, please open your browser dev tools (F12) and visit the network tab. it should show some more details in there. especially the response tab should show an actual error message. in the headers tab, you could also share the value of the cf-ray header with us and we can take a look at our logs. please be careful not to share everything you see in that network tab, as there are values visible there that allow taking over your lemmy account. specifically cookies and the jwt value; in some cases also an authorization header. cf-ray is not sensitive, it's just an identifier associated with the individual request.

feel free to pm me the cf-ray value rather than posting it publicly.

[-] IMALlama@lemmy.world 2 points 5 days ago

Thanks for the reply and apologies for getting back to you. I typed that out right before going to bed :(

Looking in developer tools, it does look like Cloudflare is what's blocking me. Here's the cf-ray value: 8f4ffe74bffc22f1-ORD

Now that I'm on a PC, I can simply open the file in an image editor, resave it without making any intentional modifications, and the upload succeeds. It makes me think that it has something to do with the image metadata (maybe EXIF?).

I'll send you a pm you a link to the unmodified image momentarily.

[-] MrKaplan@lemmy.world 4 points 5 days ago

thanks for the image. for some reason i can't see the firewall event for your attempt, but when i tried to upload the image myself i found the event. it's a rule related to a php file upload vulnerability in software we're not even using, it's disabled now.