1202
submitted 5 months ago* (last edited 5 months ago) by rxxrc@lemmy.ml to c/technology@lemmy.world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It's all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We'll see if that changes over the weekend...

you are viewing a single comment's thread
view the rest of the comments
[-] bdonvr@thelemmy.club 196 points 5 months ago

The amount of servers running Windows out there is depressing to me

[-] franklin@lemmy.world 81 points 5 months ago

The four multinational corporations I worked at were almost entirely Windows servers with the exception of vendor specific stuff running Linux. Companies REALLY want that support clause in their infrastructure agreement.

[-] Avatar_of_Self@lemmy.world 25 points 5 months ago

I've worked as an IT architect at various companies in my career and you can definitely get support contracts for engineering support of RHEL, Ubuntu, SUSE, etc. That isn't the issue. The issue is that there are a lot of system administrators with "15 years experience in Linux" that have no real experience in Linux. They have experience googling for guides and tutorials while having cobbled together documents of doing various things without understanding what they are really doing.

I can't tell you how many times I've seen an enterprise patch their Linux solutions (if they patched them at all with some ridiculous rubberstamped PO&AM) manually without deploying a repo and updating the repo treating it as you would a WSUS. Hell, I'm pleasantly surprised if I see them joined to a Windows domain (a few times) or an LDAP (once but they didn't have a trust with the Domain Forest or use sudoer rules...sigh).

[-] Semi_Hemi_Demigod@lemmy.world 15 points 5 months ago* (last edited 5 months ago)

The issue is that there are a lot of system administrators with “15 years experience in Linux” that have no real experience in Linux.

Reminds me of this guy I helped a few years ago. His name was Bob, and he was a sysadmin at a predominantly Windows company. The software I was supporting, however, only ran on Linux. So since Bob had been a UNIX admin back in the 80s they picked him to install the software.

But it had been 30 years since he ever touched a CLI. Every time I got on a call with him, I'd have to give him every keystroke one by one, all while listening to him complain about how much he hated it. After three or four calls I just gave up and used the screenshare to do everything myself.

AFAIK he's still the only Linux "sysadmin" there.

[-] Hotzilla@sopuli.xyz 7 points 5 months ago

"googling answers", I feel personally violated.

/s

To be fare, there is not reason to memorize things that you need once or twice. Google is tool, and good for Linux issues. Why debug some issue for few hours, if you can Google resolution in minutes.

[-] Avatar_of_Self@lemmy.world 3 points 5 months ago* (last edited 5 months ago)

I'm not against using Google, stack exhange, man pages, apropos, tldr, etc. but if you're trying to advertise competence with a skillset but you can't do the basics and frankly it is still essentially a mystery to you then youre just being dishonest. Sure use all tools available to you though because that's a good thing to do.

Just because someone breathed air in the same space occasionally over the years where a tool exists does not mean that they can honestly say that those are years of experience with it on a resume or whatever.

[-] uis@lemm.ee 4 points 5 months ago

Just because someone breathed air in the same space occasionally over the years where a tool exists does not mean that they can honestly say that those are years of experience with it on a resume or whatever.

Capitalism makes them to.

[-] sacredfire@programming.dev 1 points 5 months ago

Agreed. If you are not incompetent, you will remember the stuff that you use often. You will know exactly where to look to refresh your memory for things you use infrequently, and when you do need to look something up, you will understand the solution and why it’s correct. Being good at looking things up, is like half the job.

[-] uis@lemm.ee 6 points 5 months ago

Companies REALLY want that support clause in their infrastructure agreement.

RedHat, Ubuntu, SUSE - they all exist on support contracts.

[-] Rinox@feddit.it 17 points 5 months ago

I dunno, but doesn't like a quarter of the internet kinda run on Azure?

[-] atocci@lemmy.world 35 points 5 months ago

so 40% of azure crashes a quarter of the internet...

[-] Rinox@feddit.it 4 points 5 months ago

I guess Spotify was running on the other 40%, as many other services

[-] corsicanguppy@lemmy.ca 2 points 5 months ago

doesn't like a quarter of the internet kinda run on Azure?

Said another way, 3/4 of the internet isn't on Unsure cloud blah-blah.

And azure is - shhh - at least partially backed by Linux hosts. Didn't they buy an AWS clone and forcibly inject it with money like Bobby Brown on a date in the hopes of building AWS better than AWS like they did with nokia? MS could be more protectively diverse than many of its best customers.

[-] Blackmist@feddit.uk 16 points 5 months ago

I've had my PC shut down for updates three times now, while using it as a Jellyfin server from another room. And I've only been using it for this purpose for six months or so.

I can't imagine running anything critical on it.

[-] ccdfa@lemm.ee 41 points 5 months ago

Windows server, the OS, runs differently from desktop windows. So if you're using desktop windows and expecting it to run like a server, well, that's on you. However, I ran windows server 2016 and then 2019 for quite a few years just doing general homelab stuff and it is really a pain compared to Linux which I switched to on my server about a year ago. Server stuff is just way easier on Linux in my experience.

[-] conciselyverbose@sh.itjust.works 11 points 5 months ago

It doesn't have to, though. Linux manages to do both just fine, with relatively minor compromises.

Expecting an OS to handle keeping software running is not a big ask.

[-] sugar_in_your_tea@sh.itjust.works 2 points 5 months ago

Yup, I use Linux to run a Jellyfin server, as well as a few others things. The only problem is that the CPU I'm using (Ryzen 1st gen) will crash every couple weeks or so (known hardware fault, I never bothered to RMA), but that's honestly not that bad since I can just walk over and restart it. Before that, it ran happily on an old Phenom II from 2009 for something like 10 years (old PC), and I mostly replaced it because the Ryzen uses a bit less electricity (enough that I used to turn the old PC off at night; this one runs 24/7 as is way more convenient).

So aside from this hardware issue, Linux has been extremely solid. I have a VPS that tunnels traffic into my Jellyfin and other services from outside, and it pretty much never goes down (I guess the host reboots it once a year or something for hardware maintenance). I run updates when I want to (when I remember, which is about monthly), and it only goes down for like 30 sec to reboot after updates are applied.

So yeah, Linux FTW, once it's set up, it just runs.

[-] uis@lemm.ee 1 points 5 months ago

not that bad since I can just walk over and restart it.

You can try to use watchdog to automatically restart on crashes. Or go through RMA.

[-] sugar_in_your_tea@sh.itjust.works 1 points 5 months ago

I could, but it's a pretty rare nuisance. I'd rather just replace the CPU than go through RMA, a newer gen CPU is quite inexpensive, I could probably get by with a <$100 CPU since anything AM4 should work (I have an X370 with support for 5XXX series CPUs).

I'm personally looking at replacing it with a much lower power chip, like maybe something ARM. I just haven't found something that would fit well since I need 2-4 SATA (PCIe card could work), 16GB+ RAM, and a relatively strong CPU. I'm hopeful that with ARM Snapdragon chips making their way to laptops and RISC-V getting more available, I'll find something that'll fit that niche well. Otherwise, I'll just upgrade when my wife or I upgrade, which is what I usually do.

[-] uis@lemm.ee 1 points 5 months ago* (last edited 5 months ago)

I just haven't found something that would fit well since I need 2-4 SATA (PCIe card could work), 16GB+ RAM, and a relatively strong CPU.

4 SATA, 8GB RAM is easy to find. What do you need 16 gigs for? Compiling Gentoo?

Star64 for ARM and Quartz64 for RV.

[-] sugar_in_your_tea@sh.itjust.works 1 points 5 months ago* (last edited 5 months ago)

Running a few services:

  • NextCloud + Collabora (or maybe OnlyOffice, haven't decided)
  • Jellyfin - terabytes of data, need memory to avoid buffering
  • Linux package mirror - higher RAM when running updates
  • compile-farm for my personal Rust projects (e.g. CI/CD)
  • in the future, HomeAssistant and security camera streaming

Star64 for ARM and Quartz64 for RV.

You got those backwards.

But yeah, I've had my eye on those, but I think 8GB will be a bit limiting, and the CPU in both the Star64 and Quartz64 may be a bit weak. I was about to buy one, then decided to upgrade my desktop PC to a SFF PC, which meant I had some decent hardware lying around. It's mostly up and running now (still working on Collabora/Only Office), so I'll be monitoring RAM usage to see if I really need 16GB or if I can get away with 8GB instead, but I have a feeling something like either of those two SBCs will be a bit limiting.

I don't need all of those services, but I definitely want the first two, and they wouldn't need to be used at the same time (e.g. if I'm watching a movie, I'm probably not messing with big docs in NextCloud). The last one is very nice to have, but it could be a separate box (I'm gusesing Quartz64 would be plenty, though I may want to play with object detection).

[-] uis@lemm.ee 1 points 5 months ago
  • Jellyfin - terabytes of data, need memory to avoid buffering

At 8GB it won't matter going for 16. Even 4 is a lot. It's not like you watch same video many times.

  • Linux package mirror - higher RAM when running updates

Wierd. Is it not rsync? I ran mirror for my local network on 1GB SBC.

  • compile-farm for my personal Rust projects (e.g. CI/CD)

I was joking about compiling Gentoo. Might need that much RAM here.

  • in the future, HomeAssistant and security camera streaming

So, powerful CPU or hardware encoder.

Star64 for ARM and Quartz64 for RV.

You got those backwards.

My brain got SIGBUS.

then decided to upgrade my desktop PC to a SFF PC, which meant I had some decent hardware lying around.

Not bad decision. Why buy something new, when something old works too.

but I have a feeling something like either of those two SBCs will be a bit limiting.

Maybe. I wish I knew PC-style computers on ARM or RV with upgradable memory. Maybe something from SiFive? Dunno.

[-] sugar_in_your_tea@sh.itjust.works 2 points 5 months ago

I can probably run the CI/CD on my desktop, so if that's the only bottleneck, I can work around it. Or I can spin up a temporary VPS to handle it. I'll have a VPS always running to traffic over my VPN, so it could also manage orchestration (sounds like a fun project imo).

Wierd

It's just one more thing running. It's not a ton of resources, but it runs an HTTPS server and more RAM means faster updates. Since I run Tumbleweed, an update could be 1-2GB (everything gets rebuilt when a core dependency gets updated).

I could probably run it on a Raspberry Pi (have one for RetroPie), so I'm not too worried about it.

powerful CPU or hardware encoder

I probably won't need that, I'd just store the video in whatever format the cameras send in. I do have a GPU on this PC (nothing special, just a GTX 750ti), but I think I could live without that if I moved to an SBC.

The main thing I'm concerned about it Collabora/OnlyOffice, which is really RAM heavy. I'll probably only have 1-2 users at a time (me or my wife), and I just think 8GB will be a little lean.

I was super close to getting a Quartz64, but realized upgrading the motherboard and case on my desktop would be about the same price (as Quartz64 + case + PCIe SATA card) and the Quartz64 only has 1x PCIe, so I went for the upgrade (had a Ryzen 1700 w/o motherboard and a couple cases just sitting around). I'd love to go ARM, and a 16GB RAM SBC would probably convince me to switch.

[-] uis@lemm.ee 2 points 5 months ago

I'd love to go ARM, and a 16GB RAM SBC would probably convince me to switch.

Or motherboards with replacable RAM

[-] corsicanguppy@lemmy.ca -1 points 5 months ago

big ask.

Off the car lot, we say 'request'. But good on you for changing careers.

[-] conciselyverbose@sh.itjust.works 2 points 5 months ago

I really have no idea why you think your choice of wording would be relevant to the discussion in any way, but OK...

[-] zeekaran@sopuli.xyz 4 points 5 months ago

Not judging, but why wouldn't you run Linux for a server?

[-] Blackmist@feddit.uk 2 points 5 months ago

Because I only have one PC (that I need for work), and I can't be arsed to cock around with dual boot just to watch movies. Especially when Windows will probably break that at some point.

[-] uis@lemm.ee -1 points 5 months ago

Can you use Linux as main OS then? What do you need your computer to do?

[-] Blackmist@feddit.uk 2 points 5 months ago

I need to run windows software that makes other windows software, that will be run on our customers (who pay us quite well) PCs that also run windows.

Plus gaming. I'm not switching my primary box to Linux at any point. If I get a mini server, that will probably ruin Linux.

[-] uis@lemm.ee 0 points 5 months ago* (last edited 5 months ago)

I need to run windows software that makes other windows software, that will be run on our customers (who pay us quite well) PCs that also run windows.

Mingw, but whatever. Maybe there is somethong mingw can't do.

Plus gaming. I'm not switching my primary box to Linux at any point.

Unless it is Apex and some other worst offenders or you use GPU from the only company actively hostile to linux, gaming is fine.

Where did you think Microsoft was getting all (hyperbole) of their money from?

[-] neosheo@discuss.tchncs.de 1 points 5 months ago

I know i was really surprised how many there are. But honestly think of how many companies are using active directory and azure

this post was submitted on 19 Jul 2024
1202 points (99.5% liked)

Technology

60112 readers
1924 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 2 years ago
MODERATORS