302
submitted 1 year ago by alyaza@beehaw.org to c/support@beehaw.org

Yesterday, you probably saw this informal post by one of our head admins (Chris Remington). This post lamented some of the difficulties we’re running into with the site at this point, and what the future might hold for us. This is a more formal post about those difficulties and the way we currently see things.

Up front: we aren't confident in the continued use of Lemmy. We are working through how best to make the website live up to the vision of our documents—and simply put, the vast majority of the limitations we're running into are Lemmy's at this point. An increasing amount of our time is spent trying to work around or against the software to achieve what we want rather than productively building this community. That leaves us with serious questions about our long-term ability to stay on this platform, especially with the lingering prospect of not having the people needed to navigate backend stuff.

Long-time users will no doubt be aware of our advocacy for moderator tools that we think the platform needs (and particularly that we need). Our belief in the importance and necessity of those tools has only hardened with time. Progress of those tools, however—and even organizing work on them—has been pretty much nonexistent outside of our efforts from what we can see.[^1] In the three months since we started seriously pushing the ideas we'd like to see, we’re not aware of any of them being seriously considered—much less taken up or on the way to being incorporated into Lemmy.

In fact: even within the framework of Lemmy's almost nonexistent roadmap and entirely nonexistent timetable on which to expect features it has been made clear to us that improving federation or moderation on the platform are not big priorities.[^2] We have implicitly been told that if this part of the software is to improve we will need to organize that from scratch. And we have tried that to be clear. Our proposal is (and has been) paying people bounties for their labor toward implementing these features, in line with paying all labor done on our behalf—but we've received mixed messages from the top on whether this would be acceptable. (Unclear guidance and general lack of communication is symptomatic of a lot of our relation with the Lemmy devs in the past few months.)

Things aren't much better on the non-moderator side of things. The problems with databases are almost too numerous to talk about and even Lemmy's most ardent supporters recognize this as the biggest issue with the software currently. A complete rewrite is likely the only solution. Technical issues with the codebase are also extensive; we've made numerous changes on our side because of that. Many of the things we're running into have been reported up the chain of command but continue to languish entirely unacknowledged. In some cases bugs, feature requests, and other requests to Lemmy devs have explicitly been blown off—and this is behavior that others have also run into with respect to the project. Only very recently have we seen any overtures at regular communication—and this communication has not hinted at any change in priorities.

All of what was just described has been difficult to get a handle on—and having fewer users, less activity, and more moderators has not done a whole lot to ease that. We honestly find that the more we dig and the more we work to straighten out issues that pop up, the more pop out and the more it feels like Lemmy is structurally unsound for our purposes. (One such example of what we’re working with is provided in the next section.)

In summary: we believe we can either continue to fight the software in basically every way possible, or we can prioritize building the community our documents preach. It is our shared belief that we cannot, in the long-term, do both; in any case, we're not interested in constantly having to fight for basic priorities—ones we consider extremely beneficial to the health of the overall Lemmy network—or having to unilaterally organize and recruit for their addition to the software. We are hobbyists trying to make a cool space first and foremost, and it's already a job enough to run the site. We cannot also be surrogates for fixing the software we use.

PenguinCoder: A brief sketch of the technical perspective

I've said a few words about this topic already, here and here. Other Beehaw admins have also brought some concerns to the Lemmy devs. Those issues still exist. To be clear: this is a volunteer operation and Lemmy is their software; they have a right to pick and choose what goes into it and what to put a priority on. But we have an obligation to keep users safe and secure, and their priorities increasingly stifle our own.

In the case of this happening for open source projects, the consensus is to make your own fork. But:

The problem with forking Lemmy is in starting from all the bad that is inherently there, and trying to make it better. That is way more work than starting fresh with more developers. IE, not using Rust for a web app and UI, better database queries from the start, better logging/functions from the start; not adding on bandaids. A fork of Lemmy will have all of Lemmy's problems but now you're responsible for them instead.

We don't need a fork, we need a solution.

To give just one painful example of where an upstream solution is sorely needed: the federation, blocking, and/or removal of problem images.

  1. You post an image to Beehaw.
  2. Beehaw sends your content out to every other server it's federated with
  3. Federated server accepts it (beehaw.org is on their allowlist), or rejects it (beehaw.org is on their denylist)
  4. If the server accepts it, a copy of your post or comment including the images are now on that receiving server as well as on the server you posted it to. Federation at work.
  5. Mod on beehaw.org sees your post doesn't follow the rules. Removes it from beehaw.org. The other instances Beehaw pushed this content to, do not get that notice to remove it. The copy of your content on Beehaw was removed. The copy of your content on other servers was not removed.
  6. The receiving federated instance needs to manually remove/delete the content from their own server
  7. For a text post or comment that's removed, this can be done via the admin/mod tools on that instance
  8. For a post or comment including a thumbnail, uploaded images, etc; that media content is not removed. It's not tracked where in Lemmy that content was used at. Admin removal of media commences. This requires backend command line and database access, and takes about a dozen steps per image; sometimes more.

There are dozens of issues—some bigger, some smaller—like this that we have encountered and have either needed to patch ourselves or have reported up the chain without success.

Alternatives and the way forward

If possible the best solution here is to stay on Lemmy—but this is going to require the status quo changing, and we’re unsure of how realistic that is. If we stay on Lemmy, it is probable that we will have to do so by making use of a whitelist.

For the unfamiliar, we currently use a blacklist—by default, we federate with all current and newly-created nodes of the Fediverse unless we explicitly exclude them from interacting with our site. A switch to a whitelist would invert this dynamic: we would not federate with anybody unless we explicitly choose to do so. This has some benefits—maintaining federation in some form; staying on Lemmy; generally causing less entropy than other alternatives, etc. But the drawbacks are also obvious: nearly everything described in this post will continue, blacklist or whitelist, because a huge part of the problem is Lemmy.

Because of that we have discussed almost every conceivable alternative there is to Lemmy. We are interested in the thoughts of this community on platforms you have all used and what our eventual choice is going to be, but we are planning on having more surveys in the future to collect this feedback. We ask that you do not suggest anything to us at this time, and comments with suggestions in this thread will be removed.

As for alternatives we’re seriously considering right now: they’re basically all FOSS; would preserve most aspects of the current experience while giving us less to worry about on the backside of things (and/or lowering the bar for code participation); are pretty much all more mature and feature-rich than Lemmy; and generally seem to avoid the issues we’re talking about at length here. Downsides are varied but the main commonality is lack of federation; entropy in moving; questions of how sustainable they are with our current mod team; and more cosmetic things like customization and modification.

We’re currently investigating the most promising of them in greater depth—but we don’t want to list something and then have to strike it, hence the vagueness. If we make a jump, that will be an informed jump. In any case logistics mean that the timetable here is on the order of months. Don’t expect immediate changes. As things develop, we’ll engage the community on what the path forward is and how to make it as smooth as possible.

[^1]: Other administrators have probably vocally pushed for these things, but we’re not aware of any public examples we can point to of this taking place. Their advocacy has not produced results that we're aware of in any case, which is what matters. [^2]: Perhaps best illustrated by the recent Lemmy dev AMA. We’ll also emphasize that Beehaw’s admin team is not alone in the belief that Lemmy devs do not take mod tools or federation issues particularly seriously.

(page 2) 50 comments
sorted by: hot top controversial new old
[-] BananaTrifleViolin@kbin.social 19 points 1 year ago* (last edited 1 year ago)

I think you will find this conundrum on any software you switch to. FOSS is hard, and needs a big enough community of motivated people with the right skills to make a project successful. People are largely doing this work as hobbies; it's hard to fund such projects. Doable but hard.

The most obvious alternative to go for is the Reddit code base which was open source and has been forked as Saidit. This is the most likely place to find something mature enough and feature rich enough for what you may need but again whether things will progress is another conundrum as who else is maintaining or using that codebase?

Lemmy and any other project like Kbin will need people and work to get it where you want, not just suggestions and a list of requests. The problem is not a lack of interest in achieving what you want from Lemmy, it is realistically that it is a small project team with a big task on their hands and Beehaw are not it's only users.

Ultimately Lemmy may not be the software now to do what you want for your community. Federation may also not be the right thing for a community of your ethos. Maybe the simplest solution is complete defederation and build the community in an environment you can completely control, even with the limits Lemmy current provides with it's software. Come back to the fediverse when you feel the software matches the ambitions, but in the meantime build the community you want.

load more comments (3 replies)
[-] alehel@beehaw.org 19 points 1 year ago

I got to be honest. I really don't care about the federation part of Beehaw and would be quite happy to see it move to a non-federated solution. I mean, how is it even possible to comply with the GDPR when using federation?

load more comments (4 replies)
[-] Crotaro@beehaw.org 18 points 1 year ago

My biggest thanks to your continued transparency and will to keep us informed on what's going on in the upper offices of the hive.

If Beehaw eventually moves, I'll likely follow, but I fully acknowledge how scary that "restart" could be.

[-] Piers@beehaw.org 18 points 1 year ago

My only input is that any potential migration could be called "Bexit".

load more comments (3 replies)
[-] Chimaeratorian@beehaw.org 17 points 1 year ago

I do not have any solutions but want to thank and show support of the admins for the continued thoughtfulness and transparency about the issues the site faces.

I am surprised by the ELI5 on how Lemmy federation works. I guess I assumed it was somehow P2P, not a mass entanglement of duplicated content, which as mentioned is a nightmare for problematic and/or illegal content.

I don't know how the creators expected Lemmy to grow with each instance's storage and hosting costs also growing exponentially as the fediverse expands.

[-] alyaza@beehaw.org 14 points 1 year ago

I don’t know how the creators expected Lemmy to grow with each instance’s storage and hosting costs also growing exponentially as the fediverse expands.

i'm unsure if this is a "failing" of Lemmy specifically or just a general design choice of ActivityPub federation but yeah it's not ideal, i would say. beyond the huge issues if anyone posts anything that needs to be mandatorily reported it means we have like 100GB of images (mostly from federation--and mind you, while still not federating with lemmy.world which is huge) and growing. for smaller nodes of the Lemmyverse i honestly have no idea how sustainable that is.

[-] BrikoX@lemmy.zip 17 points 1 year ago

An outsider point of view

Lemmy started as a passion project and been growing slowly over years and then out of nowhere a small group of developers had to not only adjust to new influx of people, but also a barrage of ideas, suggestions and comments on what they should prioritize. I'm not sure how much experience different people here have with open source development, but the amount of changes in 3 months Lemmy developers managed to do is impressive to say the least. And there will never be open source project that manages to satisfy everyone.

Fundamentally people assume that with open source project all ideas will get implemented either by developers or if someone does the work and makes a pull request. But like with every other project the maintainers are allowed to have their own vision and not implement everything someone asks for or have different priorities or specific structure in mind.

Beehaw always tried to create something where they have complete control over everything. And that worked with federation when they were one of a few "big" instances (some people might not know but Beehaw is 21 months old, opened on 2021-11-12), but once influx of new users came in the existing tools weren't enough to have the level of control they wanted. So they clsoed the doors on new users and isolated themselves and at the same time angered a big portion of fediverse users for taking advantage of new user influx but the cutting them off from the rest of the fediverse.

I'm not sure if people checked the posts where Beehaw listed features and tools they want and a lot of them are super tailored to Beehaw vision which is not in step with federated Lemmy as a whole.

I don't think that the Beehaw vision and fediverse can coexist as they have diametrically opposed ideas.

[-] BitOneZero@beehaw.org 18 points 1 year ago* (last edited 1 year ago)

the amount of changes in 3 months Lemmy developers managed to do is impressive to say the least.

I'm trying to be nice, but two full time paid people did not manage to do much in 3 months if you mean May, June, and July regarding the Reddit API change period. It seems Rust is their main focus and in the middle of all this they decided to start a new front-end in Rust. When dozens of new front-ends were being developed by eager newcomers, including replacements like Photon that even have admin and moderation interfaces.

I’m not sure if people checked the posts where Beehaw listed features and tools they want and a lot of them are super tailored to Beehaw vision which is not in step with federated Lemmy as a whole.

I've checked their listed features and tools, and I have no idea what you are talking about. They are features that Lemmy needs. What exactly do you mean that Beehaw is "super tailored"?

load more comments (14 replies)
[-] hazelnoot@beehaw.org 16 points 1 year ago

I appreciate the transparency through all of this. It's nice to know about the problem while there's still some time to handle it. To be honest - I probably would not follow Beehaw to any new non-federated platform. Its nothing against Beehaw (I love this community and its admins) but I'm just not convinced that we could ever attain enough regular users to keep critical mass on the niche topics I follow. That's the real advantage of Lemmy - all (well-behaved) instances get to share their user base.

Unfortunately, I also understand why staying on Lemmy might not be an option. Same with forking or using another Fedi software. Either way - I trust y'all to consider all the options and make the best choice. 🐝💙

[-] Scary_le_Poo@beehaw.org 15 points 1 year ago

I wrote a big ol post that didn't post apparently. Anyway, I assume you have been looking at Postmill. If beehaw move I'm likely going to follow. Tbh without the fediverse, I would likely engage more often. I find that I dislike conversing with federated users because they come from a place with a different ethos. As a result I still have to deal with crap.

load more comments (1 replies)
[-] SomeGuyNamedPaul@beehaw.org 15 points 1 year ago

There exists ActivityPub library implementations in golang, just sayin'. It's a big lift to start anew but at least the low end protocol is there and golang is a good mature language for productivity and security.

DB is going to be to bottleneck and I'd build on ScyllaDB (or Cassandra) in a heartbeat. ScyllaDB on a single node is quite well behaved and auto-tuning, but from there these two can scale globally with scaled writes everywhere. I always architect with active/active in mind because at some point for some reason you need multiple sites even for just disaster recovery.

load more comments (2 replies)
[-] Hexorg@beehaw.org 14 points 1 year ago

Alternatives or not, I think it’d be very beneficial to document concept of operation that you want. That way you can either take pieces of these conops and tell lemmy devs what you want, or if you have your own project this will be its conops and you can guide developers towards features you need.

[-] chicken@lemmy.dbzer0.com 14 points 1 year ago

If you do choose to move away from Lemmy, would you consider implementing the same or mostly the same API in whatever you move to instead? I'm working on a client and it would be nice to be able to support connecting to Beehaw without additional work.

load more comments (2 replies)
[-] comicallycluttered@beehaw.org 14 points 1 year ago* (last edited 1 year ago)

I'm in favor of either option, honestly. A whitelist or complete migration to something else.

When I joined, I wasn't looking for a drop-in reddit replacement. I'd actually deleted my reddit account a few weeks before all the craziness went down because reddit and social news in general has a very bad habit of becoming toxic as shit.

Now, I get that I'm in a minority here. People left reddit and wanted something to replace it, but I don't know if Beehaw was ever the right instance for that specifically.

While I don't particularly care one way or the other about federation or the "Fediverse", what does worry me is whether or not the platform Beehaw migrates to is better maintained than Lemmy.

If moving to something new and relatively untested, there's a big risk that other, equally as important, development issues might crop up, especially if the dev team is relatively small.

I'm also curious to find out whether UX will be similar (eg. content aggregation with voting and whatnot) or if it'll be something closer to older forums, though I'm aware you don't want to really say anything until you're decided, so I guess answers to that can wait.

Anyway, I'll be interested to see what happens. Take your time, figure it out, and we'll see what happens over the next few weeks.

[-] admin@beehaw.org 13 points 1 year ago

...what does worry me is whether or not the platform Beehaw migrates to is better maintained than Lemmy.

We will not move from the US civil war (i.e. Lemmy) to world war 3 (i.e. apocalypse).

load more comments (1 replies)
[-] Dee_Imaginarium@beehaw.org 13 points 1 year ago

At first I thought this might be an overreaction to be perfectly honest. But I just read through some of the Dev responses in the AMA... what the fuck is the issue with removing exploding heads from join-lemmy??

The other questions on the development itself weren't awesome but wow that comment chain about join-lemmy is something else.

I could overlook the CCP support if they keep it on their instances but funneling people into an instance like that... I'm not sure how to feel about Lemmy either TBH.

load more comments
view more: ‹ prev next ›
this post was submitted on 10 Sep 2023
302 points (100.0% liked)

Beehaw Support

2797 readers
1 users here now

Support and meta community for Beehaw. Ask your questions about the community, technical issues, and other such things here.

A brief FAQ for lurkers and new users can be found here.

Our September 2024 financial update is here.

For a refresher on our philosophy, see also What is Beehaw?, The spirit of the rules, and Beehaw is a Community


This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.


if you can see this, it's up  

founded 2 years ago
MODERATORS