The article is very much off point.
- Software quality wasn’t great in 2018 and then suddenly declined. Software quality has been as shit as legally possible since the dawn of (programming) time.
- The software crisis has never ended. It has only been increasing in severity.
- Ever since we have been trying to squeeze more programming performance out of software developers at the cost of performance.
The main issue is the software crisis: Hardware performance follows moore’s law, developer performance is mostly constant.
If the memory of your computer is counted in bytes without a SI-prefix and your CPU has maybe a dozen or two instructions, then it’s possible for a single human being to comprehend everything the computer is doing and to program it very close to optimally.
The same is not possible if your computer has subsystems upon subsystems and even the keyboard controller has more power and complexity than the whole apollo programs combined.
So to program exponentially more complex systems we would need exponentially more software developer budget. But since it’s really hard to scale software developers exponentially, we’ve been trying to use abstraction layers to hide complexity, to share and re-use work (no need for everyone to re-invent the templating engine) and to have clear boundries that allow for better cooperation.
That was the case way before electron already. Compiled languages started the trend, languages like Java or C# deepened it, and using modern middleware and frameworks just increased it.
OOP complains about the chain “React → Electron → Chromium → Docker → Kubernetes → VM → managed DB → API gateways”. But he doesn’t even consider that even if you run “straight on bare metal” there’s a whole stack of abstractions in between your code and the execution. Every major component inside a PC nowadays runs its own separate dedicated OS that neither the end user nor the developer of ordinary software ever sees.
But the main issue always reverts back to the software crisis. If we had infinite developer resources we could write optimal software. But we don’t so we can’t and thus we put in abstraction layers to improve ease of use for the developers, because otherwise we would never ship anything.
If you want to complain, complain to the mangers who don’t allocate enough resources and to the investors who don’t want to dump millions into the development of simple programs. And to the customers who aren’t ok with simple things but who want modern cutting edge everything in their programs.
In the end it’s sadly really the case: Memory and performance gets cheaper in an exponential fashion, while developers are still mere humans and their performance stays largely constant.
So which of these two values SHOULD we optimize for?
The real problem in regards to software quality is not abstraction layers but “business agile” (as in “business doesn’t need to make any long term plans but can cancel or change anything at any time”) and lack of QA budget.
we would need exponentially more software developer budget.
Are you crazy? Profit goes to shareholders, not to invest in the project. Get real.
I agree with the general idea of the article, but there are a few wild takes that kind of discredit it, in my opinion.
“Imagine the calculator app leaking 32GB of RAM, more than older computers had in total” - well yes, the memory leak went on to waste 100% of the machine’s RAM. You can’t leak 32GB of RAM on a 512MB machine. Correct, but hardly mind-bending.
“But VSCodium is even worse, leaking 96GB of RAM” - again, 100% of available RAM. This starts to look like a bad faith effort to throw big numbers around.
“Also this AI ‘panicked’, ‘lied’ and later ‘admitted it had a catastrophic failure’” - no it fucking didn’t, it’s a text prediction model, it cannot panic, lie or admit something, it just tells you what you statistically most want to hear. It’s not like the language model, if left alone, would have sent an email a week later to say it was really sorry for this mistake it made and felt like it had to own it.You can’t leak 32GB of RAM on a 512MB machine.
32gb swap file or crash. Fair enough point that you want to restart computer anyway even if you have 128gb+ ram. But calculator taking 2 years off of your SSD’s life is not the best.
It’s a bug and of course it needs to be fixed. But the point was that a memory leak leaks memory until it’s out of memory or the process is killed. So saying “It leaked 32GB of memory” is pointless.
It’s like claiming that a puncture on a road bike is especially bad because it leaks 8 bar of pressure instead of the 3 bar of pressure a leak on a mountain bike might leak, when in fact both punctures just leak all the pressure in the tire and in the end you have a bike you can’t use until you fixed the puncture.
Yeah, that’s quite on point. Memory leaks until something throws an out of memory error and crashes.
What makes this really seam like a bad faith argument instead of a simple misunderstanding is this line:
Not used. Not allocated. Leaked.
OOP seems to understand (or at least claims to understand) the difference between allocating (and wasting) memory on purpose and a leak that just fills up all available memory.
So what does he want to say?
The software crysis has never ended
MAXIMUM ARMOR
Shit, my GPU is about to melt!
Yeah what I hate that agile way of dealing with things. Business wants prototypes ASAP but if one is actually deemed useful, you have no budget to productisize it which means that if you don’t want to take all the blame for a crappy app, you have to invest heavily in all of the prototypes. Prototypes who are called next gen project, but gets cancelled nine times out of ten 🤷🏻♀️. Make it make sense.
This. Prototypes should never be taken as the basis of a product, that’s why you make them. To make mistakes in a cheap, discardible format, so that you don’t make these mistake when making the actual product. I can’t remember a single time though that this was what actually happened.
They just label the prototype an MVP and suddenly it’s the basis of a new 20 year run time project.
In my current job, they keep switching around everything all the time. Got a new product, super urgent, super high-profile, highest priority, crunch time to get it out in time, and two weeks before launch it gets cancelled without further information. Because we are agile.
THANK YOU.
I migrated services from LXC to kubernetes. One of these services has been exhibiting concerning memory footprint issues. Everyone immediately went “REEEEEEEE KUBERNETES BAD EVERYTHING WAS FINE BEFORE WHAT IS ALL THIS ABSTRACTION >:(((((”.
I just spent three months doing optimization work. For memory/resource leaks in that old C codebase. Kubernetes didn’t have fuck-all to do with any of those (which is obvious to literally anyone who has any clue how containerization works under the hood). The codebase just had very old-fashioned manual memory management leaks as well as a weird interaction between jemalloc and RHEL’s default kernel settings.
The only reason I spent all that time optimizing and we aren’t just throwing more RAM at the problem? Due to incredible levels of incompetence business-side I’ll spare you the details of, our 30 day growth predictions have error bars so many orders of magnitude wide that we are stuck in a stupid loop of “won’t order hardware we probably won’t need but if we do get a best-case user influx the lead time on new hardware is too long to get you the RAM we need”. Basically the virtual price of RAM is super high because the suits keep pinky-promising that we’ll get a bunch of users soon but are also constantly wrong about that.
All of the examples are commercial products. The author doesn’t know or doesn’t realize that this is a capitalist problem. Of course, there is bloat in some open source projects. But nothing like what is described in those examples.
And I don’t think you can avoid that if you’re a capitalist. You make money by adding features that maybe nobody wants. And you need to keep doing something new. Maintenance doesn’t make you any money.
So this looks like AI plus capitalism.
Sometimes, I feel like writers know that it’s capitalism, but they don’t want to actually call the problem what it is, for fear of scaring off people who would react badly to it. I think there’s probably a place for this kind of oblique rhetoric, but I agree with you that progress is unlikely if we continue pussyfooting around the problem
But the tooling gets bloatier too, even if it does the same. Extrem example Android apps.
You make money by adding features that maybe nobody wants
So, um, who buys them?
Capitalism’s biggest lie is that people have freedom to chose what to buy. They have to buy what the ruling class sells them. When every billionaire is obsessed with chatbots, every app has a chatbot attached, and if you don’t want a chatbot, sucks to be you then, you have to pay for it anyway.
A midlevel director who doesn’t use the tool but thinks all the features the salesperson mentioned seem cool
It’s just about convincing investors that you’re going places. Customers don’t have to want your new features or buy more of your stuff because it has them. Users certainly don’t have to want or use them. Just do buzzword-driven development and keep the investors convinced that you’re the future.
Stockholders
Stockholders want the products they own stock in to have AI features so they won’t be ‘left behind’
Sponsors maybe? Adding features because somebody influential wants them to be there. Either for money (like shovelware) or soft power (strengthening ongoing business partnerships)
“Open source” is not contradictory to “capitalist”, just involves a fair bit of industry alliances and\or freeloading.
“Open source” was literally invented to make Free software palatable to capitol.
It absolutely is to the majority of capitalists unless it still somehow directly benefits them monetarily
Yeah, my favorite is when they figure out what features people are willing to pay for and then paywal everything that makes an app useful.
And after they monetize that fully and realize that the money is not endless, they switch to a subscription model. So that they can have you pay for your depreciating crappy software forever.
But at least you know it kind of works while you’re paying for it. It takes way too much effort to find some other unknown piece of software for the same function, and it is usually performs worse than what you had until the developers figure out how to make the features work again before putting it behind a paywall and subscription model again again.
But along the way, everyone gets to be miserable from the users to the developers and the project managers. Everyone except of course, the shareholders Because they get to make money, no matter how crappy their product, which they don’t use anyway, becomes.
A great recent example of this is Plex. It used to be open source and free, then it got more popular and started developing other features, and I asked people to pay reasonable amount for them.
After it got more popular and easy to use and set up, they started jacking up the prices, removing features and forcing people to buy subscriptions.
Your alternative now is to go back to a less fully featured more difficult to set up but open source alternative and something like Jellyfin. Except that most people won’t know how to set it up, there are way less devices and TVs will support their software, and you can’t get it to work easily for your technologically illiterate family and or friends.
So again, Your choices are stay with a crappy commercialized money-grubbing subscription based product that at least works and is fully featured for now until they decide to stop. Or, get a new, less developed, more difficult to set up, highly technical, and less supported product that’s open source and hope that it doesn’t fall into the same pitfalls as its user base and popularity grows.
Software has a serious “one more lane will fix traffic” problem.
Don’t give programmers better hardware or else they will write worse software. End of.
This is very true. You don’t need a bigger database server, you need an index on that table you query all the time that’s doing full table scans.
You never worked on old code. It’s never that simple in practice when you have to make changes to existing code without breaking or rewriting everything.
Sometimes the client wants a new feature that cannot easily implement and has to do a lot of different DB lookups that you can not do in a single query. Sometimes your controller loops over 10000 DB records, and you call a function 3 levels down that suddenly must spawn a new DB query each time it’s called, but you cannot change the parent DB query.
Where is this even coming from? The guy above me is saying not to give devs better hardware and to teach them to code better.
I followed up with an example of how using indices in a database to boost the performance helped more than throwing more hardware at it.
This has nothing to do with having worked on old code. Stop trying to pull my comment out of context.
But yes you’re right. Adding indexes to a database does nothing to solve adding a new feature in the scenario you described. I also never claimed it did.
but you cannot change the parent DB query.
Why not?
This sounds like the “don’t touch working code” nonsense I hear from junior devs and contracted teams. They’re so worried about creating bugs that they don’t fix larger issues and more and more code gets enshrined as “untouchable.” IMO, the older and less understood logic is, the more it needs to be touched so we can expose the bugs.
Here’s what should happen, depending on when you find it:
- grooming/research phase - increase estimates enough to fix it
- development phase - ask senior dev for priority; most likely, you work around for now, but schedule a fix once feature compete; if it’s significant enough, timelines may be adjusted
- testing phase/hotfix - same as dev, but much more likely to put it off
Teams should have a budget for tech debt, and seniors can adjust what tech debt they pick.
In general though, if you’re afraid to touch something, you should touch it, but only if you budget time for it.
That budget is the key. You have to demonstrate/convince the purse holders first. This isn’t always an easy task.
Fair. If that’s not possible, I’ll start looking for another job, because I don’t want to deal with a time bomb that will suddenly explode and force me to come in on a holiday of something to fix it. My current company allocates 10-20% of dev time to tech debt, because we all know it’ll happen, so we budget for it.
“Don’t touch working code” stems from “last person who touched it, owns it” and there’s some shit that it’s just not worth your pay grade to own.
Particularly if you’re a contractor employed to work on something specific
I get that for contractors, get in and get out is the best strategy.
If you’re salary, you own it regardless, so you might as well know what it does.
That’s why it needs to be written better in the first place
Tell me you never worked on legacy code without telling me…
Kid. Go away
You do accept that bad software has been written, yes? and that some of that software is performing important functions? So how is saying “It needs to be written better in the first place” of any use at all when discussing legacy software?
It’s not, but you’ll still hear it a lot. Funny, no one can agree on what “better” means, especially not the first person who wrote it, who had unclear requirements, too little time, and 3 other big tickets looming. All of these problems descend from management, they don’t always spontaneously come into being because of “bad devs”, although sometimes they do.
Or sharding on a particular column
Quality in this economy ? We need to fire some people to cut cost and use telemetry to make sure everyone that’s left uses AI to pay AI companies because our investors demand it because they invested all their money in AI and they see no return.
32gb+ memory leaks require reboot on any machine, and need higher level than critical.
The AI later admitted: “This was a catastrophic failure on my part. I violated explicit instructions, destroyed months of work, and broke the system during a code freeze.” Source: The Register
When I started using LLM’s, and would yell at its stupidity and how to fix it, most models (Open AI excepted) were good enough to accept their stupidity. Deleting production databases certainly feels better with AI’s woopsie. But being good at apologizing is not best employee desired skill.
Collapse (Coming soon) Physical constraints don’t care about venture capital
This is naive, though the collapse part is worse. Venture capital doesn’t care about physical constraints. Ridiculously expensive uneconomic SMRs will save us in 10 (ok 15) years. Kill solar now to permit it. But, scarcity is awesome for venture capital. Just buy the utilities, and get a board seat, get cheap, current price lock in, power for datacenters, and raise prices on consumers and non-WH-gifting-guest businesses by 100% to 200%. Physical constraints means scarcity means profits. Surely the only political solution is to genocide the mexican muslim rapists.
Accept that quality matters more than velocity. Ship slower, ship working. The cost of fixing production disasters dwarfs the cost of proper development.
This has been a struggle my entire career. Sometimes, the company listens. Sometimes they don’t. It’s a worthwhile fight but it is a systemic problem caused by management and short-term profit-seeking over healthy business growth
“Apparently there’s never the money to do it right, but somehow there’s always the money to do it twice.”
Management never likes to have this brought to their attention, especially in a Told You So tone of voice. One thinks if this bothered pointy-haired types so much, maybe they could learn from their mistakes once in a while.
We’ll just set up another retrospective meeting and have a lessons learned.
Then we won’t change anything based off the findings of the retro and lessons learned.
Post-mortems always seemed like a waste of time to me, because nobody ever went back and read that particular confluence page (especially me executives who made the same mistake again)
Post mortems are for, “Remember when we saw something similar before? What happened and how did we handle it?”
They should also include the recommended steps to take to prevent the issue again, which would allow you to investigate why the changes you made (you were allowed to make the changes, right? Manglement let you make the changes?) didn’t prevent the issue
Twice? Shiiiii
Amateur numbers, lol
That applies in so many industries 😅 like you want it done right… Or do you want it done now? Now will cost you 10x long term though…
Welp now it is I guess.
You can have it fast, you can have it cheap, or you can have it good (high quality), but you can only pick two.
Getting 2 is generous sometimes.
There’s levels to it. True quality isn’t worth it, absolute garbage costs a lot though. Some level that mostly works is the sweet spot.
The sad thing is that velocity pays the bills. Quality it seems, doesn’t matter a shit, and when it does, you can just patch up the bits people noticed.
This is survivorship bias. There’s probably uncountable shitty software that never got adopted. Hell, the E.T. video game was famous for it.
I don’t make games, but fine. Baldurs Gate 3 (PS5 co-op) and Skyrim (Xbox 360) had more crashes than any games I’ve ever played.
Did that stop either of them being highly rated top selling games? No. Did it stop me enjoying them? No.
Quality feels important, but past a certain point, it really isn’t. Luck, knowing the market, maneuverability. This will get you most of the way there. Look at Fortnite. It was a wonky building game they quickly cobbled into a PUBG clone.
I love crappy slapped together indi games. Headliners and Peak come to mind. Both have tons of bugs but the quality is there where it matters. Peak has a very unique health bar system I love, and Headliners is constantly working on the balance and fun, not on the graphics or collision bugs. Both of those groups had very limited resources and they spent them where they matter, in high quality mechanics that are fun to play.
Skyrim is old enough to drive a car now, but back then it’s main mechanic was the open world hugeness. They made damn sure to cram that world full of tons of stuff to do, and so for the most part people forgave bugs that didn’t detract from that core experience.
BG3 was basically perfect. I remember some bugs early on but that’s a very high quality game. If you’re expecting every game you play to live up to that bar, you’re going to be very disappointed.
Quality does matter, but it only matters when it’s core to the experience. No one is going to care if your first-person-shooter with tons of lag and shitty controls has an amazing interactive menu and beautiful animations.
It’s not the amount of quality, it’s where you apply it.
(I’ve had that robot game that came with th ps5 crash on me, but folding@home on the ps2 never did, imagine that)
They mainly show what’s possible if you
- don’t have a deadline
- don’t have business constantly pivoting what the project should be like, often last minute
- don’t have to pass security testing
- don’t have customers who constantly demand something else
- don’t have constantly shifting priorities
- don’t have tight budget restrictions where you have to be accountable to business for every single hour of work
- don’t have to maintain the project for 15-20 years
- don’t have a large project scope at all
- don’t have a few dozen people working on it, spread over multiple teams or even multiple clusters
- don’t have non-technical staff dictating technical implementations
- don’t have to chase the buzzword of the day (e.g. Blockchain or AI)
- don’t have to work on some useless project that mostly exists for political reasons
- can work on the product as long as you want, when you want and do whatever you want while working at it
Comparing hobby work that people do for fun with professional software and pinning the whole difference on skill is missing the point.
The same developer might produce an amazing 64k demo in their spare time while building mass-produced garbage-level software at work. Because at work you aren’t doing what you want (or even what you can) but what you are ordered to.
In most setups, if you deliver something that wasn’t asked for (even if it might be better) will land you in trouble if you do it repeatedly.
In my spare time I made the Fairberry smartphone keyboard attachment and now I am working on the PEPit physiotherapy game console, so that chronically ill kids can have fun while doing their mindnumbingly monotonous daily physiotherapy routine.
These are projects that dozens of people are using in their daily life.
In my day job I am a glorified code monkey keeping the backend service for some customer loyalty app running. Hardly impressive.
If an app is buggy, it’s almost always bad management decisions, not low developer skill.
“AI just weaponized existing incompetence.”
Daamn. Harsh but hard to argue with.
Weaponized? Probably not. Amplified? ABSOLUTELY!
It’s like taping a knife to a crab. Redundant and clumsy, yet strangely intimidating
Love that video. Although it wasn’t taped on. The crab was full on about to stab a mofo
Yeah, crabby boi fully had stabbin’ on his mind.
Anyone else remember a few years ago when companies got rid of all their QA people because something something functional testing? Yeah.
The uncontrolled growth in abstractions is also very real and very damaging, and now that companies are addicted to the pace of feature delivery this whole slipshod situation has made normal they can’t give it up.
That was M$, not an industry thing.
It was not just MS. There were those who followed that lead and announced that it was an industry thing.
I must have missed that one
Another big problem not mentioned in the article is companies refusing to hire QA engineers to do actual testing before releasing.
The last two American companies I worked for had fired all the QA engineers or refused to hire any. Engineers were supposed to “own” their features and test them themselves before release. It’s obvious that this can’t provide the same level of testing and the software gets released full of bugs and only the happy path works.
Non-technical hiring managers are a bane for developers (and probably bad for any company). Just saying.
The calculator leaked 32GB of RAM, because the system has 32GB of RAM. Memory leaks are uncontrollable and expand to take the space they’re given, if you had 16MB of RAM in the system then that’s all it’d be able to take before crashing.
Abstractions can be super powerful, but you need an understanding of why you’re using the abstraction vs. what it’s abstracting. It feels like a lot of them are being used simply to check off a list of buzzwords.
Accurate but ironically written by chatgpt
And you can’t even zoom into the images on mobile. Maybe it’s harder than they think if they can’t even pick their blogging site without bugs
Is it? I didn’t get that sense. What causes you to think it’s written by chatGPT? (I ask because whilst I’m often good at discerning AI content, there are plenty of times that I don’t notice it until someone points out things that they notice that I didn’t initially)
Not x. Not y. Z.
It wasn’t that --em dash–it’s this.
It loves grouping of 3s
Those are just the ones I noticed immediately again when skimming it, there was a lot more I noticed when I originally read it. I read it aloud to my wife while cooking the first time and we were both laughing about how obviously chatjipity it was lol.
