I suspect that quite a bit of the over-engineering I have seen was driven by the personal agendas of some of the developers: turning a boring, run-of-the-mill project into something that feels like you are working on the leading edge of technology; "I have learned all this stuff, now I am going to show I can use it"; the misapplication of design principles for large projects to small ones; resume padding. It is not done by those programmers who struggle to get anything done or by those who always do the least they can get away with out of laziness; it requires a certain level of competence and is driven by ambition.
These motivations seem to be behind many of the almost constant calls for rewrites, but rewrites with no better justification than this rarely turn out to be a good use of resources. It is particularly galling to see a rewrite that is no better than the original.
I agree with you. Only the very best developers that I have worked with advocate for principles like: no rewrites, refactor only; less is more; minimal viable product with customer feedback loops; reducing complexity; a/b testing, etc. My best work has been alongside product managers that push back on new features. I have never seen a rewrite that was functionally better than the original.
I honestly don't have any experience working with a team that produces a top-shelf title, like a popular web browser, spreadsheet, or calendar application. I've only been involved working on internal facing "intranet" business applications, because that's where the majority of the jobs are.
That being said, I wonder if what you're describing is worse for internal applications, simply because teams don't really "listen" to their users (employees)? I don't think I've ever seen a nice intranet or internal payroll application, for example. My current company, which is a very big F500 company, has one of the worst payroll and HR apps considering its size. Intranet applications tend to accumulate every departments special hidden feature without anyone keeping it in check.
Looking at Apple I feel that most of their customer facing applications and operating systems are getting worse not better. They are getting too "clever" with their user interfaces, relying on tricky unintuitive user interface gestures and hidden button clicking. I wonder if the symptoms you describe manifest themselves at Apple as UI "cleverness."
I've over-engineered plenty of software in my time. Most of the time, it was because of boredom with yet another crud app.
An anecdote about rewriting:
I've also been involved in 4 successful rewrites. The decision to scrap/start over is rarely a good one. In my case, my team had inherited a bizarre suite of apps that combined VB6/C++. It was full of memory leaks, and had never been successfully deployed in the 3-4 years of its existence. We rewrote the entire suite in a year using .NET (1.x at the time). We (developers) went onsite to a bunch of our clients and watched them work, figured out what the system really needed to do, and made it do exactly that. The result was the first successful deployment the company had had in 4 years. It also sold itself in sales meetings, because it had been designed by watching users, rather than by guesswork.
So rewrites can be done. The question is: how bad is the underlying system? And how much easier is the problem to solve in technology X vs Y?
In our case, the underlying system was atrocious, undocumented, untested, written by a COBOL programmer who did not understand VB or C++ or OOP. And the .NET WinForms stack was vastly more productive than the C++ (and even the VB6) stacks we were replacing. The decision to rewrite was pretty easy.
I believe this behavior has been dubbed "Resume Driven Development". I am sure I was guilty of it when I was a junior developer. It extends from over-engineering to also selecting the hot new technologies rather than the tried and true ones. It's actually a pretty rational behavior from a self-interest standpoint. As a manager you have to balance the needs of the projects with the needs of your staff to develop their skill sets. If you don't, they will languish and the good ones will leave. For instance, I will try to work with only 1 new technology at a time in a project, try to limit the super abstractions and over-generalization to libraries, where you might get real benefit from them someday, rather than the applications that come and go.
> It is not done by those programmers who struggle to get anything done
I've had a different experience. When I see programmers over-engineering, it is often because they would struggle to work on something that mattered.
Instead of taking on the next big feature that would make a difference to the company, they try to redesign some schema, because they've redesigned schemas before. What's more, if someone asks them what they're doing, they can hand-wave and make it sound complicated and important, and the poor non-technical managers have no choice but to believe them and let them go on not doing anything important that might challenge them.
I believe most experienced software types would agree that maintaining systems over time is a rather different skillset to building them in the first place. 'Fire and forget' is a great short-term commercial strategy, but usually a terrible operational one.
Certain (usually early) stages of a project may be time or resource rich or permit wide-ranging changes with minimal risk (client impact, downtime, live failover requirements, carefully engineered change-overs, etc.). At these times, an experienced person might opt to utilize resources to 'over-engineer' for contemporary requirements in the knowledge that it is a strong and well-reasoned long term play, while to the less experienced, non-technical project managers or stakeholders this may seem questionable.
Ultimately, every project is different but there is a 'sliding window' of project resources (particularly time and the availability of low-risk incremental change) and requirements (expanding, shifting, contracting).
A famous quote from a famous figure on migration: Those change-over things are really severe. Really severe problems. - Joseph Henry Condon, Bell Labs.[0]
Perhaps ultimately, over-engineering may be inefficient but efficient if considered in light of the alternatives: by parable, eventually including a rewrite in javascript that also reads email.
It requires you to think in many different ways. The first time you need to add a field or change a behavior and you realize that you're going to break everything running or built around the old version, your attitude towards building changes. Soon you spend more time figuring out how to make small changes that don't break anything than you do writing new features.
Suddenly versioned protocols are your friend. Suddenly you find yourself wondering how awful it really would be to have a SaveFile and a SaveFileACID. And then later you might realize that what you really wanted was SaveFileDurable (way faster), but now you're starting to feel like PHP.
And then some dependency goes and breaks half your libraries, and you become dependency averse. You vet your dependencies by the reputations of their maintainers instead of by the performance of their code. You intentionally start using slower, more awkward APIs that give you headaches today (why can't they just fix that off-by-one that they've documented so carefully?) because you know in 2 years it'll still work exactly the same. And that's more important than clean code or a 300% performance boost.
Definitely a different skill set to write maintainable systems.
The thing is the flavor of over-engineering JavaScript suffers from - bringing in frameworks/tech the project can be done without is bound to backfire at some point when (and in a ridiculously fast moving ecosystem like js we're talking 1-2 years) you can't find engineers with the skillet needed for the the "old tech"-based projects.
Most over-engineered monstruosities I've seen in my career don't come from engineers trying to uber-optimize their solutions to a sick level, but from noob/unexperienced/dumb programmers trying to get something done.
Things like: methods with just one line of code (to call another method), classes that seem to be there just to satisfy some dubious OOP pattern, too many levels of error handling which can be simplified by just removing them and catch the exception in a single place of the program (at least in exception-based languages), too much 'public' elements in a class (justified by the programmer thinking: "just in case" someone needs to access them in the future), excessive interface use due to the (clueless)need of pedantic levels of unit testing, NIH-retards creating frameworks/ORMs/libraries for every single component their software needs (I even saw once a tech lead that replaced all uses of hashtables in the company by his own implementation which, hey, let you look for keys in a case-insensitive way!), choosing low-level languages to implement non-performance-critical software, and a big etcetera.
Thing is, in my book, bad engineers are the ones that cannot get anything done (or barely do) because they are not talented/intelligent enough. These are harmless.
But what about the ones that get things done (at least their managers perceive so) but creating these enormous turds? They think they have a clue, but are just barely starting to scratch the surface, and they can harm any project a lot. For me, overengineering is not excessive optimization or quality, but excessive unnecessary complexity (when there could be an alternative, much simpler, implementation, which still complies with the requirements imposed).
> For me, overengineering is [...] excessive unnecessary complexity (when there could be an alternative, much simpler, implementation, which still complies with the requirements imposed).
I think the article relates to that though. I agree there are cases in which the complexity of objectively uneeded and will never be useful. (Reinventing the wheel, bad design, the aforementioned misapplied OOP principles and unit boundaries) but what the article stresses IMO is that it's not always easy to see if the simple implementation will comply with the requirements in the future as well. Some complexity could be justified if the speculation on the future is backed by facts.
Let's consider your first example: a method with just one line of code.
I'm not sure why you bring that up. There's nothing intrinsically wrong about a method that only has one line of code. That one line of code can, for example, contain a complex calculation and the method is an abstraction for that calculation.
Even one line of code just calling another method can be quite alright if the method is, for example, part of an adapter or facade class.
Of course this sentence seems to capture your thinking:
"excessive interface use due to the (clueless)need of pedantic levels of unit testing"
Sorry, forgot to give more details: a private method with a single line of code. That is: something which can be simply done in the caller.
> Of course this sentence seems to capture your thinking:
Yes, unit testing can be excessive too. Of course I'm more against zero unit testing than excessive unit testing, but each test is an investment, and the optimal approach is, as usual, somewhere in the middle.
> too much 'public' elements in a class (justified by the programmer thinking: "just in case" someone needs to access them in the future)
I've seen someone 15 years my senior make that mistake; then take issue when I removed the unnecessary code. He eventually had me fired.
The "just in case" thinking is often used to justify some otherwise useless OO looking pattern, which once implemented is difficult to excise, because the alternatives tend to look less "OO" and less "pattern-y".
> I've seen someone 15 years my senior make that mistake; then take issue when I removed the unnecessary code. He eventually had me fired.
You deleting his extra "thinking ahead" code was subversive behavior and you deserved to be fired. You might technically have been right that his coding preferences were suboptimal, but it's not your place to (from his perspective) randomly delete good code. As far as I can tell, you blindly followed a clean code "quick rule of thumb" and butchered the bigger picture causing actual damages to productivity in addition to waisting your developer own time.
It's more complicated. First, I only touched code I was meant to use or update. Second, the guy refused to review my code until much too late. Third, he had me suffer his daily sarcasms practically from day one, sometimes in front of the customer.
The real reason he had me fired was because I treatead him like a peer (as I was told we were), while in fact he wanted to be the boss.
My take away from this is you need to ask 'why' as often as 'what' and 'how' when developing software.
In my experience many software developers completely fail to challenge the 'why' of what they're being asked to do, and jump right to design/coding.
To be fair though, not all software developers work in an environment where they can, or feel they can, ask or challenge the 'why', but the software will only benefit if they do.
Why? is also how you resolve the XY problem[1], and asking it five times[2] is the basis of a root cause analysis, and as any parent will tell you[3] is part of raising children, and is also what you should seek to put into a commit message[4].
Ultimately, software design is a prediction of the future. You make those parts flexible where you will need it. You make other parts simple and opinionated where you need no flexibility. Like all predictions we are often wrong, which leads to either over-engineering or lack of features.
And it is incredibly hard to predict the future. So I just try to engineer things forcefully for only today's needs. And instead of future-proofing, I try to make it easily comprehendible. Make their intentions as explicit as possible. Comprehension trumps everything. Because before we can actually change code by typing it in, we need to be able to hold it in our head and mend it there. And if you can clearly see in your mind's eye what is currently there, you can see what it can become.
The above applies only when we have to 'predict' the future. Oftentimes we know the future already - there is going to be ton of traffic, we are going to need an API for this use case, we are actually going to stay in business etc.
This comment is so on point, I love working with code that has been built with that philosophy. The code remains simple and you don't waste hours trying to seperate useful complexity from bloat.
How flexible should the API be? For example, do you need multiple representations (JSON, YAML, XML, etc)? Maybe it makes sense to build it representation-agnostic, but maybe it is over engineering.
This is true, but unless you understand the problem then (which you might not since not all of us work at 50x expected load) then how do you architect it properly while also not wasting time?
It depends. How much the scope and vision of a project will vary in 2-5 years and 5+ years will vary greatly depending on the project. A small internal app for an institution or business probably should be far more cautious and simplistic than a consumer program designed to scale out as the business grows and customer needs change.
Fit for purpose seems to be the idea that the author is conveying for an end-user level, and I feel that's something that gets lost in Slack chats/at the conference table whiteboard when designing things. A program that just does what it's supposed to isn't really lacking features, it's just doing what it should. I've seen this countless times with project management just hitting feature bloat after the initial part got to the "works somewhat reliably" stage, and suddenly it's time for the feature set to expand for some reason. In the projects I was dealing with, it happened as people tried to satisfy their egos and to show off; this isn't always the reason, but it's a reason. Other times I'd assume it's just a neat feature that someone wants to work on but has no where else to put it.
I think it's just important from the get-go to determine do you really need software that evolves or do you need software that works really well at what it's supposed to do. I'm sure you can do both, but at a certain point it seems like these two directions start to pull too far away from each other.
In virtually every feature of every piece of software (or device, or product), some aspects will be over engineered, and some will be under-engineered. I think however, the point of the article was that often engineering "narcissism" gets in the way. But to some, that narcissism is just beautiful code or doing it "right". This article is hopelessly subjective, but makes its point clear enough.
There is another point though, which is separation of concerns. E.g. if I have a system with different features and some of those features require authentication, I don't want to write a separate authentication system for each feature - here, an abstraction that separates the authentication system from the actual features is really useful.
The year was 1991. My friend and I were partying and playing darts. 301, to be precise. At a certain point, I noted a subtraction error in my friend's score. I pointed it out so we checked everything and found I made some subtraction errors too. We were pretty wasted so this was no surprise.
I had a computer in the garage on my workbench. So I decided to solve this problem by writing some code. About an hour and some number of shots later, bullshitting with my friend the whole time, I had a working scoreboard program.
We continued playing darts. We were both amused that even though I was too wasted to subtract, I could still write code.
The bottom line is that whether something is "over-engineered" or not can mostly only be confirmed in hindsight.
With experience and instinct, some people are successful more often than not in predicting what level of engineering is sufficient, and they go on to become great engineers and leaders. But I am not wholly sure if this is more skill or more luck, though I am sure there is some skill.
With software, it is especially hard. Especially as you go deeper and deeper into the components. At the UI level, it might seem easy to say what is the amount of flexibility you need. When designing a common utility or infrastructure piece to be used by everyone in a large company, it is much less obvious. But I think the hardest layer is the mid-tier, which has to deal with the backend and the frontend, and support all kinds of changes and pluggability.
Not fearing change and being able to handle change is the skill required; and it can be learned from experience. An over-engineer will attempt to build a system that is too flexible and too complicated in order to mitigate all possible changes. Similarly, an under-engineered project will become of patchwork of impossible to maintain code when the changes roll in.
The skill is knowing how to evolve a project -- create new abstractions and layers -- in the face of change. To fundamentally alter the product and keep it working all at the same time.
If you under-engineer a project and you don't know how to do this eventually you'll be faced with the dreaded re-write. Throughout history how many companies and products have failed because of that. If you over-engineer you can suffer with a huge amount weight you don't need. And even then you still might get it wrong and be stuck with an architecture that doesn't work given new requirements -- which might require an even more difficult re-write.
So many engineers can't bring themselves to remove code or change their architecture; so they try and get it right the first time.
> Not fearing change, being able to handle change is the skill required
Agree x1000, I'm currently dealing with a situation stemming from this sort of thing (not being able to handle change, in software design, and philosophically with the architecture).
I also keep using the term "evolve" with respects to how to develop a good codebase. You can use design pressure to drive abstraction and logical components of a codebase. If you try to figure all that out from the start, you're going to have a bad time.
I would argue that it's more skill, but not engineering skill.
It is more a social skill; being able to read the work environment the software will exist inside, and predict the likely ebb and flow of development requirements. To have enough experience with the people involved to know how much flex a given system will have to undergo, over time.
>With software, it is especially hard. Especially as you go deeper and deeper into the components. At the UI level, it might seem easy to say what is the amount of flexibility you need. When designing a common utility or infrastructure piece to be used by everyone in a large company, it is much less obvious.
There's a relatively straightforward approach to this that works well - simply don't build in flexibility until it's explicitly required or it's very, very obvious it will be required.
When you're engineering mid-tier and back end the requirements can be most easily ascertained by doing the upper levels that depend on them first and working your way down.
I'd estimate that 9 out of 10 times when an engineer tries to predict and pre-empt future requirements they fail and that failure is often expensive not only in wasted time but in useless extra code to confuse maintainers (and new space for bugs to grow in).
The problem in my experience is that the pain of unanticipated change, or poor early engineering choices is felt acutely by programmers (sorry, I am going to call them that). It is really annoying to work with ill suited code. It is demoralizing to rewrite code. Programmers want to write code that lasts and hate when the code they wrote is unsuitable, as a matter of professional pride.
But the pain of over engineering is felt less acutely. Delays and confusion are often not realized in the moment. When the negative feedback comes it often comes from the 'other', a manager or another engineer who doesn't understand.
Pain of over engineering might be more hidden and difficult to point at, but in my experience it hurts much more. At least with under-engineered code you know what to do... But will you ever be able to remove unnecessary abstractions?
> Over-Engineering is when someone decides to build more than what is really necessary based on speculation
This is so wrong, in the context of software at least.
Over-engineered software has layers of abstraction that add nothing to the function or form of the product the costumer sees - it is over-engineering for the sake of ideological notions about design patterns etc.
As for functionality delivered to the client, software very, very rarely gets over-engineered - in reality it almost always gets delivered late and with core features still in development. The idea that there is all this software being delivered on time that not only meets 100% of all the original requirements with exacting quality and perfection, but is somehow delivering in gross excess to the point of giving customers more than they need is pure fiction.
I don't remember who it was. Might have been Michael Feathers, might have been Jeff Atwood, could have even been Slava Pestov.
One of them said words to the effect, "people over-engineer because they're so afraid of making a bad decision that they try not to make any decision at all." Make a decision. If it's wrong, own it. Develop OTHER coping mechanisms for being wrong. It's good for the ego, probably good for the soul.
The fights I have gotten into with myopic people over the years have largely come down to them putting a bunch of energy into what should have been reversible decisions, and treating irreversible decisions in a cavalier manner (eg, trying to 'get it over with').
Learn to tell the difference. Be bold with the reversible ones. Put off the irreversible ones if you actually can - but know that deciding to do nothing IS often a decision. Like deciding to add security or localization 'later'.
how does that go for refactoring? when I refactor code I'm typically making a decision that we have now pinned down the functionality of the code, and we are going to make the structure reflect that functionality clearly and directly while stripping off incidental complexity that came from exploring the design space.
I remember a bloody configuration mechanism in a .net gui a several years ago where to add a simple configuration entry you had to add 3 different properties in the app.config, add three fields in three different interfaces and three different implementations, add three properties in some unrelated "adapter" class and add finally the code to access them where needed.
Needless to say that it took me more time to add a bloody property than to implement a simple feature.
And 99% of the time I forgot a simple step and discovered that it didn't work only at runtime.
This is the essence of over-engineering in my opinion.
I agree this may be subjective, but aren't we crossing the line when the 'how' becomes more important than the 'what' (i.e. the purpose of the code?)
Following this principle code example 2 (i.e. Javascript after deduplication) would be the sweet spot. It fulfills the requirements without in the most obvious way, at a good level of abstraction (the requirements did not make any of the three pictures specific -- defining a separate function for each actually goes beyond the requirements, while obscuring the higher-level goal).
All of the subsequent examples focus on using the features of the platform rather than clearly showing the purpose of the code, i.e. unnecessarily obfuscate it.
There is a good argument for using
$('.image').on('click', clickImage);
instead of attaching three click handlers clickImage[ABC] and hardcoding those in HTML, which probably explains why jQuery is so popular.
OTOH I agree that using jQuery for animation doesn't add anything, particularly now that CSS Animation is omnipresent. My initial though on the problem was "Can I do this in pure CSS?", not to go to javascript. And since it turns out you can't, the logical next question is why the desired behavior is actually desired, instead of e.g. just making the image a thumbnail and a clickable link to the full-size image. So really the entire question is an example of overengineering, requiring an absurd amount of technology for a feature you aren't even sure you want.
The real downside of over-engineering is that time expenditure is non-refundable. If you build a lightweight v1, you still have the option to invest additional time into something more "robust". If you choose the opposite, you don't have the option to get your time back by making the product less engineered.
Even worse, if your under engineered product is causing problems, you'll almost certainly notice it (rather quickly too). If you over engineer, you may never realize it.
Nope, for example if an application is designed without dependency injection because it is more "lightweight" it is a pain to introduce it later.
If you designed it from the start to have clear dependencies and with an IOC container then you'll have a much easier time in changing it when there are new requirements.
As you can see the boundary between good engineering and over engineering is pretty thin.
And it obviously depends from the context.
In a toy application probably you can survive without a container.
In a real world project you are going to kill yourself without it (been there, seen that).
Designing for dependency injection is mostly about passing objects around as a parameters instead of using globals / singletons.
It's in big code bases where the tempatation to use a singleton is greatest, because no one wants to refactor all the function signatures to add the extra parameter.
Making every part of your system an injected thing can lead to massive configuration points where there is only one possible choice for much of what is configured. That is annoying and I fail to see how it really helps extensibility.
Oddly, I think the problem is often lack of a global namespace. Consider how you pull in the "parseFoo" method. In languages with a global namespace, you just use it. You can put a declaration saying the method has to exist, but no need to import it from anywhere. That is a build time concern, not a write time one.
Even better, changing the implementation does not require changing the user site, at all.
Namespaced languages have to use factories and abstract factories to get the same flexibility. (Or dependency injection.) I don't even think they're bad ideas, just find it interesting how much effort it takes to change which method I'm actually calling.
I agree with you about fractories often being stupid replacements for free-functions. I am thinking of more dynamic things: e.g. a handle to a network connection should be passed around instead of singletoned.
But even if dependency injection has all the costs you claim for it. Those costs are at their smallest when the system is small. So I still say xypud is wrong: you should do injection by default and when it gets too cumbersome then you think about how to streamline the interfaces.
To be honest...I have not thought hard on this. I expected my post to get down voted, as I figured i was taking a frowned upon view. To be more fair, I suspect I am wrong.
In practice, I agree with your point. I'd rather do some small costs up front. I'm curious to explore the designs more. I think in systems where you have independent programs working together, a lot of this solves itself. The trend today is to make things one big program. (Not a new trend.)
When it is done properly it has practically no costs and only benefits.
If you start using the container in more than one place, introducing a ServiceLocator pattern, using providers instead of automatic factories then you will pay an high cost.
But in these cases you are doing it wrong.
Doing it properly it helps you to design the whole application correctly and adds flexibility and modularity for free.
As I said, while on a toy project it is a negligible advantage, in a real world project it is an invaluable plus.
Well you correctly diagnose the biggest problem with DI: it's all or nothing.
Only nowadays all usually includes significant 3rd party library code and everybody is stuck trying to figure out how to get their DI magic in place when said 3rd party code doesn't cooperate.
Why do we never talk about how this over-engineering stuff relates to skill development? It's the same, no matter if you look at soccer stars, chess, or programming.
In the beginning people are clueless and just try to make things change and take a horrible amount of time for it. Then they start to develop some patterns. Then they see patterns from other people and choose one as their one true hammer that will hammer all the nails. Then he spents an awful time arguing why X and Y are nails. Then he sees that not all methods apply all the time and starts to balance them with each other. And finally if he gets that far, he will start to develop his real own, efficient methods.
So where is the over-engineer in this? Not in the psychopath section but in the hammer-and-nail section. He developed far enough to be effecient in some ways but still waists quite some time on trying to make all the problems to apply to his solution. That's normal. That's also about the skill level that you SHOULD have when you leave a university type of education. In soccer that's the point where the usual children club stuff ends and they start to compete in serious competitions, maybe one age level below the one that gets public attention. You want to put people on this skill level against real tasks to realise that not everything is a nail.
So instead of always blaming people for being in this level we should appreciate it. Our university education seems to work for that kind of job quite well. And then we should continue to put these people in the right environment to develop further and provide value to their teams despite the things that they still lack.
I'm trying to think of examples of overengineering in software and am having trouble from my own experience and extensive reading.
I can think of languages like C++ that facilitate what I would use a more derogatory term for like autoerotica. The site's example, taking Javascript seriously, almost always falls into this category.
In something like an automobile, using boxed steel everywhere would make the thing sturdy as hell at the expense of fuel economy and less objective things like aesthetics. A bridge design might be too expensive, or a material too heavy to facilitate the architectural vision. Etc.
In software this term is used where the software is actually under-engineered, like a shoddy deployment without any consideration of consequences. Technology like a message queue or distributed database are the victims of poor understanding or oversold claim. Almost always the real problem is clueless management and a symptom of fad driven architecture.
So going back to the materials example.. What software wouldn't benefit from end to end back pressure? Circuit breaker fault management? Memory safety? Formal proof? None of these things should have any bearing on the user, and there are a lot of companies could afford to figure out how to make one or more of these things have minimal effect on front side of SDLC with very positive effects on the long tail. Software is almost to fluid and inexpensive to be over-engineered.
In the cases I've encountered it, the term over-engineering in the context of software development is more often used to refer to building in allowances for future functionality rather than for making the software more robust. Although I have seen the latter as well - for example massively increasing the complexity of some part of a distributed system to eke out a theoretical eight 9 of availability.
The concept of over-engineering has been co-opted more recently by Agile zealots as a major bogeyman to the point where I've seen people trot it out to argue against structuring code to account for needs that are several weeks out because "It's not in THIS sprint! YAGNI!", with the predictable result that several weeks later it creates more work to restructure the code and when invariably there isn't time for that, leaves a worse architecture in the long-run as the needed functionality is bolted on.
In my experience YAGNI-abuse happens when people apply it at the wrong level.
For example, people might ignore some rule of the architecture in a rush to produce some feature. This sounds pragmatic: the feature worked, so that archtectural rule must be overengineering right?
Then you ask why this particular feature was needed now at this point the project timeline, and why we had to throw aways fripperies like error detection to get it. Then I rarely hear clear answers.
Over-engineering just means implementing something that is more advanced/complex than what was needed to get the job done (in the short term).
It's not necessarily a bad thing in a technological sense, but it's bad for business.
If people can build an extremely powerful, complex system which works well and is stable, then that's great.
Some engineers will stay away from any system which they regard as 'complex' or which doesn't strictly adhere to their technical philosophies about simplicity.
Personally, I think that if enough engineers invest their energy into building a highly complex system, I think eventually they can make it work.
You can just look at operating systems; they are extremely complex but so much human effort has been invested in them that they have become quite stable and have become essential to the functioning of our society.
This is how I feel about newer systems like Docker and Kubernetes - I think that together, they are extremely complex. When Docker started out, I thought it was crazy, I didn't see the utility value; but enough fools got involved that it eventually grew into something meaningful. I think that the offerings we have now with Docker, Kubernetes, Swarm, Mesos, etc... have become extremely compelling.
I think often it's regarded badly in engineering when you have a single component that does a lot of different things (often, that's what people often call over-engineering) - But look at the general-purpose CPU; we didn't need to invent it; people could have been satisfied with building specialized electronics for every kind of machine that just did one job well (no programming needed) but this would have slowed down innovation (because manufacturing circuit boards is expensive and time consuming).
Making generic components that do a lot of different things CAN be extremely useful when you look at the big, long-term picture. It just has to be done well and needs to be given the time to stabilize.
Idea:
As a writer, I want a platform to write a story that allows me to focus on the content instead of the formatting
Solution 1:
Build the platform using JavaScript. Do user research to come up with the best design. Hire a marketing team to receive a lot of attention and consecutively feedback on the platform. Hire data scientists to analyze the site traffic and decide which feature to add or not... in the end, through small cycles of feedback, continuously improve the platform and spend millions of dollars to come up with something very good after some time
Solution 2:
Use medium.com, because sometimes a rather complex idea can have a very simple solution.
For motorists, over-engineering is how the engines worth buying are those which are expected to last 10 years even though they are only warranted for five years.
It's the way many bridges having a posted weight limit of 10tons can handle a 12ton truck without incident.
If you go to all the trouble of actually engineering something to begin with, might as well include some effort into making it stand the test of time while performing beyond expectations. One of the most fundamental features of true engineering is a "margin of safety".
Traditionally this is what increases the chances of your product having that most useful feature; "worth buying".
The specs are the part I found most interesting.
1) First the specs say 3 images, then they say 2. if you are dealing with a manager that can change the specs like that you might want to over engineer just a bit to cover 'arbitrary number of images'. yes I know this is probable just an error by the writer of the article, but it demonstrates something that happens in the wild.
2) Communication is the missing piece in this story. when a developer is given a task they should ask questions about future requirements. "Is it always going to be 3 images?", "will the images always grow by the same ratio?" and so on. the way to not solve imaginary problems is to talk with the stakeholders.
I think 'over-engineering' is too broadly used. Clearly some solutions are over-engineering, but most are built by skilled and experienced engineers who know that requirements change / evolve. In this (common) scenario, not predicting it - and creating the necessary 'over-engineered' solution - will lead to a failed project. Alternatively, predicting the wrong thing to 'over-engineer' is equally problematic.
Without these gambles, the project will often fail too.
Managers are always suspicious that engineers are doing the wrong things. Living under this cloud is just part of being an engineer.
Granted, my experience is more with hardware than with software, but getting a design "just right" can be costly in analysis and testing, and there's a palpable concern that if you aim low rather than high, a substandard product will get pushed out the door. It can be exponentially more costly to correct a weak design late in a project than to over-design it early: Subsystems and schedules become more inter-dependent, the closer you get to launch.
I can give you a good design in six months, or a bad design in twelve. ;-)
The take home point I had from this article was that the requirements of will determine what you need to build. If this is a startup with minimal resources, the requirement is that you need to create something fast to see if you can solve a problem. At that point you might take those learnings, throw away the code and build a more extensible and robust version 2. The problem I often see is that people over engineer based on a set of requirements which assumes that the problem you are trying to solve is the right problem to solve.
In my experience, over-engineering is driven by a fear of refactoring. For whatever reason, refactoring is a process that is both culturally and technically undernourished.
1. Make sure engineers are engrossed in the business case. This sets proper scope.
2. Don't generalize something ahead of time, only after you do the same thing twice. The architecture should support easy refactoring, and there should be certain discipline about actually doing the refactoring the second time around.
There are many challenges to overcome with both of these, but it will be worth it.
In my experience we live in an era of near-insurmountable technical debt, and gratuitous complexity as the quick-and-dirty solution is always slap on another leaky abstraction.
Under-engineering is the problem for anything that's supposed to last.
My definition of Over engineerkng is that you try to cover many edge cases which are not belong to the current system scope, or you try to concern system performance improvement cases which are uncertain.
Let me tell you a story about companies that often confuse thinking about the best solution with over-engineering...
You get hired by this startup with great ideas. They have really hit their niche nicely and are getting more and more customers. You build a great throwaway prototype and customers and the mighty CEO love it. You ask when you are going to rebuild the prototype, since it's a bit of a mess... The hipster CTO says that there is no need, that you are the greatest dev ever and that he will get a tattoo with your name.
The first change request comes in. Easy. It's still straight-forward to make some quick and dirty changes here and there. You call it a day and your hipster CTO throws a party to celebrate the amazing job.
After change request 10, you feel like slowing down drastically. The mental model you could fit in your brain until now, is too complex. You can't touch all the involved parts of your application anymore or it would cost you a lot of time. You get away with some if/else statements to fix the edge cases. You feel happy again, you did it! You call it a day, your hipster CTO pats your back and gives out some beers.
After change request 20, the system is a mess. Different layers of the application talk directly to each other, thus creating painful coupling. Your abstractions are either non-existent or all in the wrong places. You have no CI, because that would have been over-engineering the test suite. Your local tests take half an hour, because making them run in parallel would have been over-engineering. The time for documenting the project was spent hunting quirks somewhere in the darkest places of your application. You don't even try to touch some of the code you wrote just a couple of months ago. "There be dragons" is, next to "TODO", the most used comment you find in the codebase.
So after many days, the job isn't done yet. The mighty CEO gets nervous and starts running all over the place, giving you some "leadership". He says that there is a deadline not to be missed or the universe will collapse. The hipster CTO even has to skip his yoga-classes to meet the deadline. After many extra hours and even more energy drinks, you get the change request done. The hipster CTO just smiles at you, but no beer this time.
There is 10 new bugs but you don't know about them yet. Soon you will, because people will start complaining about that last change you did and the bugs you introduced. By this time, you are sick and tired of it all. You hate yourself. You feel guilty. Problem is, you don't even have the time to think about how tired you are. Request 30 is going to arrive soon...
But hey!, don't forget that you HAD TO ship it quick, not think about solving the problem! Abstractions? Over-engineering! Designing for resilience? Over-engineering! Small, independent services? Over-engineering! Staging system? Over-engineering! All these thoughts come to your mind everytime. Yet, you are so tired, you fall asleep and try to forget it all...
WAKE UP! THERE IS A BUG IN PRODUCTION!!!
You need to rush. In the middle of the night, your messy apartment gets illuminated by the blueish light of your laptop. You don't even care anymore.
The End
---
Appendix
By change request 40, your soul committed suicide.
By change request 50, the company was spending 80% of their time doing bug fixes and keeping their self-build house of cards running.
By change request 60, there was this tiny competitor who gathered together a nice little team of skilled and motivated engineers. The CTO would give them time to think about how to solve the problems best. Soon they entered the market at an amazing pace. All competition went out of business. You lost your job. You have no motivation. You feel lonely.
By change request 70, which never actually happened, that tiny competitor sees you as a valuable asset to their company. Since they are growing, they need people like you, with experience and skills. You still feel like the worst developer ever, but you take the job. People respect you. The software was designed following best practices. It is stable, people iterate quick. You can take your time to think about the best solution. People share their ideas and with each iteration they make it better.
> The software was designed following best practices. It is stable, people iterate quick. You can take your time to think about the best solution.
Except that the messy project you were working on before sounded like a one man project. And at every new change request you delivered, working by yourself, a pretty big chunk of value.
Now, in the nicely engineered project, you work in a team. The team is iterating in sprints that require a number of developers two or more weeks of work to deliver the tiniest feature. Yes, iteration is quick, but it doesn't really deliver much value. Every time the new CEO requests something even slightly different from the exact functionality that was developed so far, the project architect's answer is that this will not fit in the current set of features, and it might be possible, if ever, after another series of sprints that will take multiple months to complete.
And the funny thing is that, while in your one man project you were considered a lone cowboy working on a relatively simple system, the architect of the current project (that delivers more or less the same functionalities but requires 5 or 6 times the time and resources to maintain) is considered a demi-god "because the project is so complex". Because of course the CEO measures the complexity of the project not but what it delivers, but by the amount of resources consumed by it. And while the CEO of the previous company was annoyed when a new feature required more than a week to implement all by yourself, now whenever the new architect says "we need more resources for that", this actually increases his power and consideration in the company, as it is taken as a sign of the complexity of the project.
Nice story. I understand the point of the article, that there is such a thing as over-engineering. But as your story illustrates (and we can all attest to), there is also such a thing as under-engineering. I think the really insightful commentary is on which techniques typically fall into which category.
Over-engineering is a tough thing to discuss and there's never going to be a suitable answer anyone agrees upon. It is also a huge topic with a scope far beyond this textbox or a medium article.
Anyway, among many of the big issues here are so many things are contextual - the problem domain, stakeholders, programmer abilities, business climate, etc. It's been my experience that you cannot apply judgement to a solution without understanding as many of the variables involved as possible. Shifting these variables even slightly can change everything.
For example, say I build a feature A at company Y and then I change jobs next week and to company Z which is much smaller. Company Z wants the same feature I just built at company X, but of course I can't steal it outright. Building the same thing at company Z requires different engineering decisions, given the different climate and probably different development team, programming language, and so on hypothetically. Even if I could just copy the solution, at company Y it might be perfect, while at company Z it might be considered over-engineered. It's a combination of both objective and subjective judgements.
As for the concrete implementations of features, I often am driven crazy by both over and under engineering. I worked with several people in positions of power above or horizontally to me that would routinely shoot down many well-engineered features and implementations by other people because they were, "Too complicated." I began to realize this was often shorthand in most cases for, "I know nothing about this, my ego or political stance is threatened, I lack talent, and/or I am simply a moron." Of course I also hit plenty of actually over-engineered things that were indeed exactly this, but in many of these cases it was obvious. Where it is less clear is anything of decent scope, challenge, subjectivity, or other hard to quantify measure. I think people can look at something like the FizzBuzz and see that the parody versions using OO 10 design patterns with abstract factories are obviously over-engineered, while it gets much harder in the context of a big project.
And this leads to another problem. Complexity is not over-engineering, nor vice-versa always. Nor is scope, or code size, or any of these related concepts that come to mind. A very small amount of code can be incredibly complex, a large amount of code can be simple, and so on. Either can be over or under engineered. Just look at math - some of the most useful tools are very difficult to understand, while others are incredibly simple with tons of layers, nuances, and so on. Programming is not that much different in this regard. We've all seen things done in a tiny amount of code we would have never thought of and maybe can't even understand. Likewise, we've encountered huge amount of code that someone complains about but is written clearly, thoughtfully, and cleanly. These's a good related talk on some of these ideas by Rich Hickey if I recall.
Another issue is that it is hard to quantify both knowing what and when justifies substantial engineering efforts. As I mentioned, context plays a big part, as does intuition and experience (in relation to ability, not just pure years). These are all nearly impossible to quantify, and yet must play a large role in cases where large engineering decision making. A lot of people see things they don't understand or are new to them and just throw out the over-engineered card. Likewise, a lot of people love pulling out their favorite tool (ex: SQL, Framework of choice) and try to use it as a golden hammer. Anything else is "over-engineered." These things can lead to over-engineering or at the very least, poor engineering.
Over-engineering is also something that is relative. It is really hard sometimes to work with someone else and to get them to understand especially those hard to quantify things such as intuition. Programming can also often highly be about ego and ownership. People will not hesitate to point at something not written by them as over-engineered because it was not done how they would do it.
Game programming for me again comes to mind as an area where this is a constant pain. The inexperienced game programmer will both under and over engineer things like you've never seen. First, they'll try to use their favorite language to write a game. They'll then do things when they learn a technique like OO programming and try to apply it everywhere, to everything. "A monster? Obviously I should make a monster base class, then subclass this for an Orc, then subclass it again for an Elite Orc." Everything must be designed around the tools they know instead of the problem. The minute this person does enough in their career to get any real authority or a real job, they take this unchecked mentality and try to impose it on the people around them. I see the same thing in web application and desktop programming. Someone learns lambdas, functors, currying, design patterns, IOC, monads, etc. and they must apply them to any situation that triggers this recognition. The "engineering" then comes from the solution and tools, perhaps the XY problem in many cases. Usually these people also fail to consider the aforementioned contextual concerns.
I know in game programming this happened often where some programmer would come in and replace some lengthy looking code that made heavy use of arrays and manual for loops. They'd then go in and replace it all with vectors, iterators, and all kinds of heavier stuff. They never considered that the code was engineered that way for performance or for portability or backward compatibility or some other reason. Moreover, they didn't actually know that the seemingly "simpler" solution was actually making things as a whole more difficult by pushing out these concerns to be fixed or handled elsewhere (ex: making up performance somewhere else).
On a personal note, I am often annoyed by the over-engineered finger pointing. There are so many things like XML messes and 200 steps to setup something for a simple purpose that are easy enough to agree on as over-engineered. Most of these come down to asking yourself, "What problem am I trying to solve" and perhaps repeating those steps aloud. If you feel ridiculous saying them, especially to another person, something might be wrong. But if it turns out that the problem you are trying to solve really does require something that is not a sound-bite, the problem explanation will also often have to be long and cannot be skirted around without ignoring the problem.
Too often I see things in this industry thrown out there to "fail hard," to be an MVP, to test the market, to disrupt, to fill a gap. That's all fine, but make sure you are solving the problem and not creating new ones like security, performance, misleading people, defrauding investors, or worse because you thought engineering a proper solution is too hard or takes too long. Find the sweet spot and make informed decisions. I recommend another talk about Rich Hickey (sorry two by the same person, just memorable talks) about Hammock-driven development and thinking before acting. If more people did that, it could swing either way and maybe we would have less short-term "stuff" but better long-term solutions, less over-engineered messes, and things that actually are worth building on top of as bases. As it stands now, this industry is littered with poorly engineered solutions we were stuck with because of both over and under engineering.
> A famous comic using the attempt of building a tree as the analogy of what is wrong in each part of the traditional development lifecycle
The project in the comic actually appears to be building a tree swing. I'm sure this was just an oversight on the author's part but if anything it reinforces the point the comic is making.
Author here. You are right, I did not know the name of the thing because English is not my primary language. Thanks for teaching me that, I have edited the picture caption to show the correct description of the image. Thanks!
"Tree swing" seems fine to me (native speaker as well), the commonality of all pictures is it is a swing, set up in a tree (albeit incorrectly or poorly in many cases). If I google "tree swing" it is clear that this is not a particularly rare way to refer to this specific sort of swing.
Yeah, it's worth noting that "tree swing" isn't a common idiom; I only meant a swing that's set up on a tree. I just didn't want to call it a "tire swing" though because it only took that form in one panel. Maybe "rope swing" would be the best name for it?
So? The author didn't 'engineer' medium in order to distribute this post. Maybe he was already familiar with the site, or maybe he finds it the easiest to use?
Medium does far more than serve this one article. Over engineering would be to make sure that it only uses the minimal subset of Libs to deliver each article. Instead they publish a bit of cruft with each article, but why would they change it'll never add to their bottom line.
When it's not one person or group doing it but the responsibility is widely distributed in space (many people in many places) and time (evolving complexity) we tend to just accept things as "that's just how it is" and not even try to challenge it. Which of course has a practical aspect, since it is about the hardest thing of all to try to change this kind of stuff.
For example, when I compare my Atari 1040 STFM and what I could do with it with the progress in hardware and software and what I can do now I can only conclude that hardware and software advances greatly outpaced the actual "customer-facing" abilities of the system. Sure I can do much - much - more now, I'm talking about scale comparison. So all those much-greater-scale advances disappear somewhere between the internals of the system and what the user gets in the end. But saying exactly where or even changing it would be an incredibly challenge, and given the continuously changing landscape practically impossible.
Not sure how that applies in this context, "overengineering" after all is exactly not what I describe since that's a word meant to used for a deliberate design process, but that's my thoughts for the above comment and the issue raised by the OP comment about page size and complexity on Medium.
The argument makes perfect sense. Nobody can use the theoretical swiss army knife pictured in the article - it's too wide, and ridiculous.
In the graphicon of over-engineering taking over users and enveloping them, pushing design out of the way, it implies that the result is hostile to users, and pushes design out of the way.
But where we read this is an example of the person selecting the "knife" in question out of every single blogging platform on the planet. So, the swiss army knife pictured is inaccurate, it's a straw man. Secondly, the author clearly finds the design appropriate enough to use, and a great match for their user stories.
So whereas the central thesis in the article says:
> "small, elegant, and successful systems tend to be plagued with feature creep due to inflated expectations".
The author is not using a "small, elegant, succcessful system".
Perhaps the swiss army knife pictured is the best knife on the planet, and not one of the worst. This fundamentally undercuts the point the author is making.
I believe the comment I was replying to was saying one should not use Medium because it is an "over-engineered blogging platform." I was not arguing for creating or using theoretical swiss army knifes.
These motivations seem to be behind many of the almost constant calls for rewrites, but rewrites with no better justification than this rarely turn out to be a good use of resources. It is particularly galling to see a rewrite that is no better than the original.