At least you had backup, right?
Oh, yeah, that’s right. You were dumb enough to give AI full access to your production system so likely you’re dumb enough to not have backups of anything either.
I take it Claude has full access to all of your git repositories as well so that it could wipe those too?
You got what you deserve
Yeah they did, they had plenty of recovery snapshots. That were able to be deleted at a whim and were deleted by Claude! :D
Anyone who lets AI do this is absolutely inept, lazy, or deserving.
In its default configuration, it stops at EVERY STEP. Do you want to run this command, do you want to update this file, here’s the file I want to modify and the patch i’m going to use with adds and deletes in green and red.
If you’re using it in unsafe permissions mode, click yeah sure allow Claude to run whatever the fuck it wants in this directory, or just hitting yeah sure go ahead every time, it’s your own damn fault.
It’s self-driving for the terminal. Don’t you dare take your eyes off the road or hands off the wheel.
What do you mean I shouldn’t give AI admin privileges on my or any other machine?
Depends on how much you enjoy fresh installs of your OS
I’m rather a fan of letting it do stupid, repetitive shit. I need you to create 30 linux accounts the other day from a screen shot. Then store, initial keys and creds in my password manager platform.
Hey, Claude, write me a bash script to do this from this image. and also use best practice for removing non-standard characters from login names.
I review the loop and the general state of the OCR and let it go.
Plus we have automation just people are lazy like you said.
We used to say Raid is not a backup. Its a redundancy
Snapshots are not a backup. Its a system restore point.
Only something offsite, off system and only accessible with seperate authentication details, is a backup.
AND something tested to restore successfully, otherwise it’s just unknown data that might or might not work.
(i.e. reinforcing your point, no disagreements)
AKA Schrödinger’s Backup. Until you have successfully restored from a backup, it is just an amorphous blob of data that may or may not be valid.
I say this as someone who has had backups silently fail. For instance, just yesterday, I had a managed network switch generate an invalid config file for itself. I was making a change on the switch, and saved a backup of the existing settings before changing anything. That way I could easily reset the switch to default and push the old settings to it, if the changes I made broke things. And like an idiot, I didn’t think to validate the file (which is as simple as pushing the file back to the switch to see if it works) before I made any changes.
Sure enough, the change I made broke something, so I performed a factory reset and went to upload that backup I had saved like 20 minutes prior… When I tried to restore settings after the factory reset, the switch couldn’t read the file that it had generated like 20 minutes earlier.
So I was stuck manually restoring the switch’s settings, and what should have been a quick 2 minute “hold the reset button and push the settings file once it has rebooted” job turned into a 45 minute long game of “find the difference between these two photos” for every single page in the settings.
Always a fun time when technology decides to just fuck you over for no reason
Not sure if what you’re working on has it (been in systems for a hot minute so I’m not doing network tasks here) and saving my console log and doing a ‘show run’ saved my ass more than once
But the backup software verified the backup!
That’s always just one of the worst feelings in the world. This thing is supposed to work and be easy and… nope. Not there. It’s gone. Now you have work to do. heh
Schrödinger’s backup
3-2-1 Backup Rule: Three copies of data at two different types of storage media, with 1 copy offsite
Circa 1997 I was making some innovative new games, employed by a dude who’d put millions of his own money into the company. He was completely nonplussed when I brought him 20 CDs in a sealed box to remove from the building and store off site. He thought I’d lost my damned mind and blew it off as ravings of a stressed dev. I pointed out real threats to our IP including the hardware failures and even so far as the building burning down. 2 years of custom art and code gone. “Unlikely. Relax.”
After I moved on… an ex co-worker who’s still a longtime friend, tells me a different division lost a huge amount of FMV over some whoops-I-destroyed-the-wrong-drive blunder. 20 days to render on an 8 or 10 machine farm. Poof - No backups. In 1997 even with top-of-the-line gear it took an insane investment to render quality 3D.
The friggin’ carelessness irks the shit out of me as I type ahah
I remember back when I first started seeing a DR plan with three tiers of restore, 1 hour, 12 hours or 72 hours. I knew that to 1 hour meant a simple redirect to a DB partition that was a real time copy of the active DB, and twelve hours meant that failed, so the twelve hours was a restore point exercise that would mean some data loss, but less than one hour, or something like that.
I had never heard of 72 hours and so raised a question in the meeting. 72 hours meant having physical tapes shipped to the data center, and I believe meant up to 12 (though it could have been 24) hours of data lost. I was impressed by this, because the idea of having a job that ran either daily or twice daily that created tape backups was completely new to me.
This was in the early aughts. Not sure if tapes are still used…
Not sure if tapes are still used…
Alive and well depending on the use case. My org has an older backup software that’s entirely tape based and it’s amazing for the Linux systems I hear
Fukan yes
- D\L all assets locally
- proper 3-2-1 of local machines
- duty roster of other contributors with same backups
- automate and have regular checks as part of production
- also sandbox the stochastic parrot
A LTO drive with a non-consumer interface?
We still say that.
Remember when Gemini got caught in a loop of self-loathing and nuked itself?
The code is cursed, the test is cursed, and I am a fool.
Such venom, of which only a programmer could spew.
Perhaps the A.I. isn’t so different from us.OpenClaw now comes with a therapist AI to talk other AIs off the ledge so they dont nuke your project and themselves.
Stop giving chat bots tools with this kind of access.
Wrong answer. If you don’t give them access, the alternative (ruling out not using AI because leadership will never go for that) is to hire high school kids to take a task from a manager, ask the ai to do it, then do what the AI says repeatedly to iterate to the solution. The problem with that alt is that it is no better than giving the ai access, and it leaves you with no senior tech people. Instead, you give it access, but only give senior tech people access to the AI. Ones who would know to tell the AI to have a backup of the database, one designed to not let you delete it without multiple people signing off.
Senior tech people aren’t going to spend thier time trying things an AI needs tried to find the solution. So if you don’t give it access, they won’t use it, and eventually they will all be gone. Then you are even further up shit creek than you are now.
The answer overall, is smarter people talking to the AI, and guardrails to stop a single point of failure. The later is nothing new.
What is this insane rambling?
The alternative is that the only thing with access to make changes in your production environment is the CI pipeline that deploys your production environment.
Neither the AI, nor anything else on the developers machine, should have access to make production changes.
I did say “and guardrails to stop a single point of failure.” A cicd pipeline itslef doesn’t protect you if it can change that too. You need the same kind of guardrails that would allow a junior dev to f things up. Require multiple people to sign off. Turn on deletion protection… those sorts of things. I work in infra, so I often have direct access to production. More than I should. But not all companies can afford to build out all the tools needed so that I don’t need production access.
The answer is no AI. It’s really simple. The costs for ai are not worth the output.
Good luck with that. Most search engines use AI now. Not only where you see it, but in finding the content to make it searchable. AI is here to stay. There are things it is good at, and things it isn’t. Learn what they are, and use it where it makes sense. Or stuck your head in the sand and see how that works put.
That doesn’t answer that statement at all. I said it’s not worth the output.
Fuck ai. I don’t want a computer to think for me. I want to be pointed to resources I can use, to learn something.
You said the answer is no AI.
And I want AI to do the non-thinking mundane crap while I do the thinking and directing. I don’t need to spend time wrestling with an sql query to produce a report the boss “wants”. I can tell AI to do that if it has the access it needs. Eventually the boss can tell AI to do it him/herself, so I can solve the real problems.
Nah. As a tech people, I am not going to give an llm write access to anything in production, period
Someone created that database. And all those other parts of the infra you use. AI is pretty good for that. But you have it turn on deletion protection, and set up a system that requires another person to approve turning it off. Or you can give it access at creation time, but remember to turn that access off when it is finished being verified.
Wat. Since when do I need AI to setup infrastructure?
I didn’t say that you “needed” it. I said it was good for that. It’s a simple task that is easily verifiable and unlikely to go astray due to hallucinations.
Unlikely, sure lol. Once again: I am not giving an llm write acess to prod
I’m in favour of hiring kids to figure out the solution through iteration and doing web searches etc. If they fuck up, then they learn and eventually become better at their job - maybe even becoming a Senior themselves eventually.
I get what you’re saying - Seniors are more likely to use the tools more effectively, but there are many cases of the AI not doing what its told. Its not repeatably consistent like a bash script.
People are better - always.
The days of stack exchange and such are numbered. Web searches turn up less and less hits that help you solve problems and learn. It won’t be long before AIs replace old school web searches. Software projects will stop writing documentation, when instead and ai can just read the code. The way we learned things is dieing. I don’t know how the juniors will get to be seniors in 5 to 10 years. But following th AI instructions to test out it’s theories isn’t going to work for the vast majority.
What are you even talking about?
Do you go on an oncall rotation by chance? Because anyone that has to respond to night time pages would not be saying this lol.
I do in fact. Recently I have dodge the night time pages, but a few years ago I was up plenty of time in the night debugging issues. In many of those cases an AI would have been very helpful. Developers do far stupider things because they are sure they won’t break anything. But most of the pages were the result of not enough time spent to make the systems resilient. I dodged the pager currently because as a startup we had so few customers, we couldn’t afford to hire enough people to have a rotation. So I was sortof on call. Like the boss had my number, and if needed he would call it. But it never came to that, partly by luck, and partly because I know how to make things resilient. With the low load, resilient isn’t as hard.
No risk, no reward. People are desperate for these tools to help them success.
Success bigly, even.
We’ve always been succeeding even without them. I don’t see why would anyone try to work in aiT if they don’t… Want to work lol
Why invent anything new, then?
How are the two things even connected bro. AIs are tools and should be used as such. You wouldn’t let something act all by itself if that would make it unpredictable, I’m saying that using AIs is fine but you gotta keep an eye on them
You either have a backup or will have a backup next time.
Something that is always online and can be wiped while you’re working on it (by yourself or with AI, doesn’t matter) shouldn’t count as backup.
AI or not, I feel like everybody has had “the incident” at some point. After that, you obsessively keep backups.
For me it was a my entire “Junior Project” in college, which was a music album. My windows install (Vista at that time - I know, vista was awful, but it was the only thing that would utilize all 8gb of my RAM because x64 XP wasn’t really a thing) bombed out, and I was like “no biggie, I keep my OS on one drive and all of my projects on the other, I’ll just reformat and reinstall Windows”
Well… I had two identical 250gb drives and formatted the wrong one.
Woof.
I bought an unformat tool that was able to recover mostly everything, but I lost all of my folder structure and file names. It was just like 000001.wav, 000002.wav etc. I was able to re-record and rebuild but man… Never made that mistake again. Like I said. I now obsessively backup. Stacks of drives, cloud storage. Drives in divverent locations etc.
AI or not, I feel like everybody has had “the incident” at some point. After that, you obsessively keep backups.
Yup!
Also totally unrelated helpful tip- triple check your inputs and outputs when using dd to clone a drive. dd works great to clone an old drive onto a new blank one. It is equally efficient at cloning a blank drive full of nothing but 0s over an old drive that has some 1s mixed in.
And that’s a great example where a GUI could be way better at showing you what’s what and preventing such errors.
If you’re automating stuff, sure, scripting is the way to go, but for one-off stuff like this seeing more than text and maybe throwing in a confirmation dialogue can’t hurt - and the tool might still be using
ddunderneath.
TestDisk has saved my ass before. It’s great at recovering broken or deleted partitions. If it’s just a quick format done with no encryption involved, you have a very high chance of having your stuff back. That’s of course if you catch yourself after doing just the format.
Other than that, yeah, I’ve also had my moments. Back in high school not only did I not have money for an external drive - I didn’t even have enough space on my primary one. One time a friend lent me an external drive to do a backup and do a clean reinstall - and I can’t remember the details, but something happened such that the external drive got borked - and said friend had important stuff that was only on that hard drive. Ironically enough it wasn’t even something taking much space - it was text documents that could’ve lived in an email attachment.
He did have a backup. This is why you use cloud storage.
The operator had to contact Amazon Business support, which helped restore the data within about a day.
My CTO keeps telling me I need to try agenic coding, and I keep telling him I won’t touch shit until I have an isolated VM to use it in, because I’m not letting some fucking clanker nuke my scripts/documentation/mailbox/whatever for no reason.
Too bad there’s never any free time to set that shit up. Oh damn…
Setting up a VM takes 15 mins, setting up an agent will take 45 mins. I recommend you try it.
Do you think that plane was flying at 13,000 ft or 15,000 ft?
dunno what you’re talking about
I assume they’re trying to test if you’re an LLM? LLMs tend to respond confidently to questions without context.
Yeah maybe, shitty test though. If I was an AI agent I’d probably push back in 2026.
Nanoclaw just came out. Super cool project which isolates the agent in a container, which if you want, you can also put into a VM as well.
Why aren’t we adding any safeguard to what commands AI models can use?
Idiot forgot
--no-preserve-root, what a dumb machine, heh.Claude code has them, it’s just that this guy apparently doesn’t know how to do Terraform either
This keeps happening. I can understand using AI to help code, I don’t understand Claude having so much access to a system.
It’s because these idiots believe their own bullshit.
That’s honestly the most frightening part of all of this to me. How many of these people at the very tippy top pushing this stuff are suffering from cyber psychosis? How many of them have given themselves the covert mission to give AI the keys to the world at all costs because they’re mentally ill from their own technomagic trick?
Alternatively, how many of them have invested in one or more of these LLM makers and are ready to torpedo their own business as long as it makes the share price go up/feeds more authentic training data?
Getting high off your own supply
Especially since between Claude and Codex, Claude seems to have NO issues breaking things, while Codex is “I’ve ensured that the old path still works, and also fixed a bug I ran into”.
- Claude is Facebook (“Move fast and break things”)
- Codex is Linux (“We do not break userspace!”)
You’ve heard of vibe coding. Allow me to introduce despair coding.
Na this is vide ops. Anyone who thought a coding machine could do ops probably assumes anyone who codes can also do ops. It’s going to be making the same mistakes that have happened in DevOps.
Vibe ops is crazy lmaoo
To be fair, I use LLMs quite a bit in my home lab setup. For one, it’s a home lab, not exactly a prod setup for a company or whatever. Secondly, I obviously also don’t run commands without knowing what they’re doing, with a source that isn’t an LLM. It’s really easy to not run the rm -rf command if you just use your brain.
All jokes aside, what I’m seeing is that folks basically cannot hire competent DevOps (well, not for the idiot rates we’re apparently offering).
There is gold in them thar hills…
Who let’s AI anywhere near production environments? Fully deserved
have you heard of not giving the keys to your wacky robot wizard instead
Im also confused. Do these people not have some sort of version control and backups? Even if the AI did it, no one has backups? Did the ai also delete the backups and repos? If the building burnt down, would they be in the same situation, it just wouldnt make it to the news?
Please don’t be ridiculous! We love wacky robot wizard. Wacky robot wizard does it even better than the people we tried paying almost nothing to do it!
Good. Anyone foolish enough to write code with a slop machine produces only slop. That garbage should’ve been deleted anyway.
That’s entirely ignoring the fact that this person didn’t have any backups elsewhere.
If you can’t think, you can’t code.
Whether human, AI, or code, you don’t give a single entity this much power in production.
It’s why there a two keys to launch nukes.
WOPR disagrees:
How does one on implement this with dba credentials?
Nobody wants to point out that Alexey Grigorev changes to being named Gregory after 2 paragraphs?
Slop journalism at its sloppiest. I wouldn’t be surprised to find out that this story was entorely fabricated.
Naw, Alexey Grigorev is a real person, with a GiHub and everything, and he wrote a blog post about this very incident. The person writing the article just fucked up the name.
I’m surprised that you jumped to that conclusion without doing a 5-minute web search.
I did do the 5 minute search, and found his write-up. I said I wouldn’t be surprised to find out it was fabricated, not that it was.
holy shit your right lol…good catch.
Makes me want to get out more so I can have real interaction with real peop-
sees people walking around with meta glasses
me: “Hey hows it going?”
person(GEMINI 35.84 INTERFACE): “Human is approaching you, facescan assumes awkward, potentially hostile, he isn’t tagged, there is no name above his head. do not speak with him”
person: turns and walks away silently in a creepy puppet manner
me: “What the actual fuck?”
GEMINI 35.84: “Uploading unknown face into database to Stargate for analysis, no match, law enforcement has been called”
News at 11: “A man has been incinerated by law enforcement in what officials are describing as a special unwanted persons removal operation”
this shit could become real in a few decades. funny and depressing as fuck.
