There's the old joke that metalheads are nice people pretending to be mean, while hippies are mean people pretending to be nice.

Investment giant Goldman Sachs published a research paper

Goldman Sachs researchers also say that

It's not a research paper; it's a report. They're not researchers; they're analysts at a bank. This may seem like a nit-pick, but journalists need to (re-)learn to carefully distinguish between the thing that scientists do and corporate R&D, even though we sometimes use the word "research" for both. The AI hype in particular has been absolutely terrible for this. Companies have learned that putting out AI "research" that's just them poking at their own product but dressed up in a science-lookin' paper leads to an avalanche of free press from lazy credulous morons gorging themselves on the hype. I've written about this problem a lot. For example, in this post, which is about how Google wrote a so-called paper about how their LLM does compared to doctors, only for the press to uncritically repeat (and embellish on) the results all over the internet. Had anyone in the press actually fucking bothered to read the paper critically, they would've noticed that it's actually junk science.

I cannot handle the fucking irony of that article being on nature, one of the organizations most responsible for fucking it up in the first place. Nature is a peer-reviewed journal that charges people thousands upon thousands of dollars to publish (that's right, charges, not pays), asks peer reviewers to volunteer their time, and then charges the very institutions that produced the knowledge exorbitant rents to access it. It's all upside. Because they're the most prestigious journal (or maybe one of two or three), they can charge rent on that prestige, then leverage it to buy and start other subsidiary journals. Now they have this beast of an academic publishing empire that is a complete fucking mess.

This has been ramping up for years. The first time that I was asked to do "homework" for an interview was probably in 2014 or so. Since then, it's gone from "make a quick prototype" to assignments that clearly take several full work days. The last time I job hunted, I'd politely accept the assignment and ask them if $120/hr is an acceptable rate, and if so, I can send over the contract and we can get started ASAP! If not, I refer them to my thousands upon thousands of lines of open source code.

My experience with these interactions is not that they're looking for the most qualified applicants, but that they're filtering for compliant workers who will unquestioningly accept the conditions offered in exchange for the generally lucrative salaries. It's the kind of employees that they need to keep their internal corporate identity of being the good guys as tech goes from being universally beloved to generally reviled by society in general.

AI systems in the future, since it helps us understand how difficult they might be to deal with," lead author Evan Hubinger, an artificial general intelligence safety research scientist at Anthropic, an AI research company, told Live Science in an email.

The media needs to stop falling for this. This is a "pre-print," aka a non-peer-reviewed paper, published by the AI company itself. These companies are quickly learning that, with the AI hype, they can get free marketing by pretending to do "research" on their own product. It doesn't matter what the conclusion is, whether it's very cool and going to save us or very scary and we should all be afraid, so long as its attention grabbing.

If the media wants to report on it, fine, but don't legitimize it by pretending that it's "researchers" when it's the company itself. The point of journalism is to speak truth to power, not regurgitate what the powerful say.

You can tell that technology is advancing rapidly because now you can type short-form text on the internet and everybody can read it. Truly innovative stuff.

This is bad science at a very fundamental level.

Concretely, we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent. Within this environment, the model obtains an insider tip about a lucrative stock trade and acts upon it despite knowing that insider trading is disapproved of by company management.

I've written about basically this before, but what this study actually did is that the researchers collapsed an extremely complex human situation into generating some text, and then reinterpreted the LLM's generated text as the LLM having taken an action in the real world, which is a ridiculous thing to do, because we know how LLMs work. They have no will. They are not AIs. It doesn't obtain tips or act upon them -- it generates text based on previous text. That's it. There's no need to put a black box around it and treat it like it's human while at the same time condensing human tasks into a game that LLMs can play and then pretending like those two things can reasonably coexist as concepts.

To our knowledge, this is the first demonstration of Large Language Models trained to be helpful, harmless, and honest, strategically deceiving their users in a realistic situation without direct instructions or training for deception.

Part of being a good scientist is studying things that mean something. There's no formula for that. You can do a rigorous and very serious experiment figuring out how may cotton balls the average person can shove up their ass. As far as I know, you'd be the first person to study that, but it's a stupid thing to study.

At some point in the last decade, the ~~ostensive~~ ostensible goal of automation evolved from savings us from unwanted labor to keeping us from ever doing anything.

source

I'm becoming increasingly skeptical of the "destroying our mental health" framework that we've become obsessed with as a society. "Mental health" is so all-encompassing in its breadth (It's basically our entire subjective experience with the world) but at the same time, it's actually quite limiting in the solutions it implies, as if there's specific ailments or exercises or medications.

We're miserable because our world is bad. The mental health crisis is probably better understood as all of us being sad as we collectively and simultaneously burn the world and fill it with trash, seemingly on purpose, and we're not even having fun. The mental health framework, by converting our anger, loneliness, grief, and sadness into medicalized pathologies, stops us from understanding these feelings as valid and actionable. It leads us to seek clinical or technical fixes, like whether we should limit smart phones or whatever.

Maybe smart phones are bad for our mental health, but I think reducing our entire experience with the world into mental health is the worst thing for our mental health.

"I gave an LLM a wildly oversimplified version of a complex human task and it did pretty well"

For how long will we be forced to endure different versions of the same article?

The study said 86.66% of the generated software systems were "executed flawlessly."

Like I said yesterday, in a post celebrating how ChatGPT can do medical questions with less than 80% accuracy, that is trash. A company with absolute shit code still has virtually all of it "execute flawlessly." Whether or not code executes it not the bar by which we judge it.

Even if it were to hit 100%, which it does not, there's so much more to making things than this obviously oversimplified simulation of a tech company. Real engineering involves getting people in a room, managing stakeholders, navigating conflicting desires from different stakeholders, getting to know the human beings who need a problem solved, and so on.

LLMs are not capable of this kind of meaningful collaboration, despite all this hype.

I know this is just a meme, but I'm going to take the opportunity to talk about something I think is super interesting. Physicists didn't build the bomb (edit: nor were they particularly responsible for its design).

David Kaiser, an MIT professor who is both a physicist and a historian (aka the coolest guy possible) has done extensive research on this, and his work is particularly interesting because he has the expertise in all the relevant fields do dig through the archives.

It’s been a long time since I’ve read him, but he concludes that the physics was widely known outside of secret government operations, and the fundamental challenges to building an atomic bomb are engineering challenges – things like refining uranium or whatever. In other words, knowing that atoms have energy inside them which will be released if it is split was widely known, and it’s a very, very, very long path engineering project from there to a bomb.

This cultural understanding that physicists working for the Manhattan project built the bomb is actually precisely because the engineering effort was so big and so difficult, but the physics was already so widely known internationally, that the government didn’t redact the physics part of the story. In other words, because people only read about physicists’ contributions to the bomb, and the government kept secret everything about the much larger engineering and manufacturing effort, we are left with this impression that a handful of basic scientists were the main, driving force in its creation.