So my manager today asked me if I could stay later when there’s broken things in prod, and then today his star dream employee yolo’ed a full stack change into prod without review. It’s fucking massive and implements new API endpoints, touches >20 files. Many of the diffs are too large to render in the browser.
It’s almost comical, but something immediately broke.
Most of my day, I’m digging through code to identify bugs created from this shit, just to get a stealth merge midday.
I kind of don’t know what to do.
I kind of don’t know what to do.
Rollback and go home
I had a colleague who pushed a load of half cocked broken nonsense to a repo i was owner of. Blatant emoji comments from AI and all. I just reverted it because I was under a load of pressure to get it out and on ASAP.
He messaged me 6 months later asking me if I “deleted his code”. I told him I reverted it because he pushed a load of broken bullshit to main and swanned off for 6 months.
He was like, “oh…” then i realised. This guy doesn’t even know what git revert is. Had to explain it.
What the actual fuck.
I say as long as Ai exist, real programmers will always have a job and are not replaceable. YOLO Vibelords are replaceable, but only real programmers have the skill to go through the shit. That’s the most secure job ever if you ask me.
Don’t provide bailouts for neither your manager nor your colleague. Highlight as far up the chain as you can the damage their behavior is having on the business.
I would say no to staying back. The employee that broke it needs to fix his code, at the very least roll the change back.
At the next meeting or post mortem bring up why code can be pushed to prod without review. Your git forge should be able to block merging to prod by a single person.
Yeah. I log off at EOD and don’t check messages. It’s not my families fault he lets his employees vibecode shit into prod. The Vibelord is the main problem, but others do similarly stupid things.
Adding to that, about the “stealth merge”, couldn’t it be just the rollback? Don’t change what works, as the saying goes.
And expanding on the meeting or post mortem idea, take this time not to fix the code, but to make an essay showing what broke, how it broke, and how it could be avoided in the future. Also preferably not letting your tone getting acid or those hearing from the company could instinctively refuse to listen to you even if your points are solid.
If only there was some way of reverting the merge commit and redeploying.
Do NOT under any circumstances fix it. The other guy needs to and you need to be clear he broke it and how
I came here to say exactly this. The offending dev and your manager aren’t going to understand the severity of issue until he spends days trying to find the bugs. Not your circus, not your monkeys.
I would immediately fire somebody with such a mentality. Letting bugs live in production just to prove your point, that’s borderline criminal. I get you’re not happy with the vibecoder’s job but there are other, smarter ways to deal with this than sabotaging the company.
These bugs shouldn’t have reached production. This is why you do reviews in chunks.
Probably very true (again, it depends on the product/company).
Gunna be honest, if you really feel that way you’d fire the vibe coder. I don’t give a shit if it’s broken ifs due to the negligence of management and other employees.
Poor planning on your part does not constitute an emergency on mine. And ofc this varies based on the service as all things do. But it’s rarely important enough that I’d stay over. Like if lives where at risk or it’s a “bring the company down but” maybe I’d stay. But there would be hell to pay for whoever broke it later. Either that or the company would loose me as an employee
By that logic wouldn’t you be firing the vibe coder for not rolling back the commit?
Fallback and fix is the best solution.
Maybe. Context is not sufficient (mistakes can happen, intention counts) but breaking prod may be bad depending on the product type.
Other ways such as?
I dunno, human things like communication and letting people know what’s wrong here and how not to let that repeat. If it’s pathological then maybe that work environment is not fit for you.
You need SRE concepts. First, if you break it then you fix it; in a system where anybody can make a change, it’s the changer’s responsibility to meet service objectives. Second, if your boss doesn’t find that acceptable then they need to appoint a service owner and ensure that only the owner can make changes; if the owner breaks it then the owner fixes it. Third, no more than half of your time should ever be spent fixing things; if something is constantly broken then call a Code Yellow or Code Red, tell your service users that you cannot meet your service levels, and stop working on new features until the service is stable again.
Under no circumstances, ever, should anybody stay late. There should only be normal business hours, which are best-effort, and an on-call rotation which is planned two months in advance. Also, everybody on call should be paid hourly minimum wage on top of salary for their time.
Roll it back and go home.
Pushing to prod without review and breaking the running application is a resume-generating event in many companies. In many others it’s not even possible because of programmatically enforced policies.
If your company’s response is not to prevent or dissuade it but to have other people work overtime to fix the mess then that’s a major management fail.
Try to educate your boss about best practices. This incident should give your arguments some more weight.
Deployment to prod should not be something a developer can do by themselves; a proper CI/CD system can be configured so that prod can only be deployed to by people with an appropriate role (product owners or lead devs if your company doesn’t have POs).
If you don’t have such a system, make it an explicit policy: Only Steve the lead dev (or someone specifically appointed by him while he’s absent) can push to prod; if anyone else does it they get invited to an uncomfortable meeting with Steve. If they do it again the meeting will be with HR.
But seriously, you should lobby for a proper CI/CD system (if none is present) and for the system to be configured so that a) you can’t merge to the main branch without a code review and b) deploying to prod only works from main and with explicit approval by a PO/lead dev. That should stop most of the shenanigans.
Roll it back and go home
If your manager won’t hear anything negative about Vibelord, polish up your resume. There’s nothing you can do to change the culture of your team if the boss isn’t on board with it.
Not sure what kind of person your boss is, but if you want to try to win him over, try to figure out what might sway him. If he’s technically illiterate, talk to him about best practices (like, I dunno, code reviews) and cite reliable sources and data if you think that would help. Avoid calling out Lord Vibington, the main thing is to put a picture into your boss’s head of what this could/should look like. He might not realize that these issues are preventable.
The goal here is to get your boss to take more ownership of the team’s culture, and start insisting on preventative measures. Mr Viberator will either have to conform, or there will be increasing friction between him and the boss.
Explain to your manager that if an employee can merge code without reviews, he should expect breakages on regular basis.
Propose that your team can meet to talk about how to make the process better, so no breakages occur.
Mention personal responsibility and knowledge - it’s a good practice that a person who wrote the latest code changes either helps either is responsible for making the service working again, if there are issues - specifically because they might have some idea what’s broken.
If your manager will ignore it, I’d start looking.
Ask to be paid by the hour
Changes that big need to be broken down into smaller PR-sized steps. To do that you need an actual plan of how to get from here to there. Bro is swaggering like a senior then coding like a junior.
What to do? Well now you have evidence that AI tools are not yet good enough to write code that doesn’t need human review before deploying to prod. Chat with whoever is in charge of Process, and get some team guidelines in place to prevent unreviewed code going to prod, and unreviewable PR’s being submitted for review.
You render diffs in the browser?
And no, let the yolo fix the things he broke, or he will never learn to be a responsible coder.
browsers are quite capable of rendering text, even images!
But they’re bad at it. Especially in performance and amount.
Edit: drag&drop kernel.log in your brower, if you don’t believe me.
Not true. Browsers are definitely faster handling text than your average terminal emulator without GPU capabilities, they just use more resources.
That said, GitHub diff viewer specifically is a sluggish mess. But that’s not because of the browser.
What, you think the usual text editor renders all lines just because? No, they load the lines that are currently visible + some more for scrolling. At least the ones that don’t choke on a mere 100k lines. And they don’t need GPU for this.
But webbrowser engines just aren’t made for this, but for DOM rendering.
So does a browser. Modern browsers don’t even need to receive the whole HTML to start rendering things.
There’s a lot of engineering effort put into browsers, much more than in terminal emulators because they need to do much more than just rendering text.
Ok, maybe i have a slight bias, because editors in-browser are usually a system-in-system(-in-system) JS mess.
GitHub and similar tools show PR diffs in the browser.
I’d just talk to the guy, clarify HE needs to stay late not me, then log off. If he’s nice about it I’d consider ataying but from your description, he’s anything but.
What you gotta do in situations when someone is beingan asshole is spark a meeting with the higher ups so they can see the BS you deal with and how it’s none of your business that prod was messed up.
