Agile Development or Bad Design? (AKA Chickens with Helicopters)

Please compose all posts in Emacs.

Moderators: phlip, Moderators General, Prelates

Masuri
Posts: 536
Joined: Sun Jul 22, 2007 8:23 pm UTC

Agile Development or Bad Design? (AKA Chickens with Helicopters)

Postby Masuri » Wed Nov 21, 2007 8:32 pm UTC

Well, it depends on what you want to do, I guess. It also depends on your mindset. In the workplace, there's a great deal of engineer vs. programmer drama. You find the engineers in the infrastructure space - OS admin, SAN engineer, system DBA, infrastructure application work (administering scheduling applications, backup dudes, network, telephony, etc). You find programmers in development land. They are writing applications, churning out ETL work, wrangling data and what not. Sometimes, however, you find programmers in the infrastructure space and this leads to suffering.

I refer to this as the Chickens in Helicopters theory.

Scenario: A chicken wants to cross the road. He's standing there watching the cars scream by at 65 miles per hour. How is he going to get to the other side without becoming roadkill?

Programmer: Well, we could build a suspension bridge over the road so that the chicken doesn't get hit. Or we could airlift him across, all we need is a helicopter.... but we don't have a helopad. So, first we should build the helopad on the side the chicken is on - then another one on the other side of the road so that the helicopter can land to pick him up and land to deposit him. Wait! Do we even have a helicopter? Well, maybe we should build one! They are handy to have around and other chickens may need to cross this road in the future so we could have the helicopter waiting at the helopad for chickens. But what if there's a chicken on the OTHER side of the road and he wants to cross to THIS side? We need to set up a rotating schedule of helicopter lifts from one side of the road to the other! But we don't want the helicopter to fly endlessly from one side of the road to the other unless there really are chickens to transport. So we need a queuing system to indicate whether chickens are there or not, and if so, to have the helicopter come get them! Let's get to work!

Engineer: Why does it need to cross the road?

If you dream of outlandish avian conveyances, take software engineering. If not, then computer science is for you and we'll see you on the infrastructure side.

canuckotter
Posts: 47
Joined: Mon Nov 12, 2007 4:27 pm UTC

Re: Computer science vs. software engineering?

Postby canuckotter » Wed Nov 21, 2007 8:42 pm UTC

fortyseventeen wrote:EDIT: Note that I don't have anything against coding/hacking (not to be confused with agile development). It's a wonderful way to become familiar with or discover new concepts and frameworks, but trying to pass it off as professional/academic work is shameful.

Quick and dirty hacks in the right places are just as professional as a perfectly-engineered elegant solution, and over-engineered nonsense is just as unprofessional as ugly hacks in the wrong place.

You'd be amazed how often the best solution really is something that looks to an outsider like a hack job. In construction, you don't build a tool shed to the same standard you'd use for a library, and ordinary residential homes have vastly less strict requirements than hospitals. Even within the same category, location makes a difference -- someone building a home here in Ottawa that meets California's earthquake-resistance code is wasting time, money, and resources. Engineers here just don't bother making the same kinds of calculations to calculate earthquake resistance, because there's no point. Does that mean that Ottawa-based architects and engineers are doing an inferior job compared to their California equivalents? Of course not.

Think about it this way... Imagine that as a tiny part of the overall program, you have a list that'll need to be sorted. You could spend a week building a super-fast optimised sort for that list, or you could spend ten minutes hacking in some existing code that's reasonably efficient but not great. Which would you use? If you chose either option, you're wrong. A good software engineer would know that before he could answer the original question, he'd need to know how many items the list is likely to hold and how critical performance of the sort is going to be. If it's a short list (say, under a hundred items) and you spend more than ten minutes to hack in some existing reasonably-efficient sort, you're wasting time. Even if it's a long list, if the sorting's able to be done in the background (perhaps while also waiting for an HTTP request to return) then you should also probably use the 10-minute solution. The only time you should look at the week-long solution is if you're going to have a long list to sort and performance really is critical. (And even then, you should see if there's a way you can get away with not sorting the whole list all at once, or maybe even not sorting at all...)

A good software engineer knows that there are times when careful design and forethought are required, and puts a lot of energy and attention into those situations when they come across them. The rest of the time, a good software engineer knows to just bang stuff together and make it happen and not worry about it too much, because for most stuff, "good enough" actually is. Knowing when to do a quick hack is just as important as knowing when not to, because the more time you spend engineering random crap, the less time you have for engineering the really important stuff.

Of course, a good engineer also knows how to make sure that their assumptions are tested so if they missed a situation where they should have been putting in extra effort, it'll be discovered and they can go back and fix it. It's one of the advantages of being a software engineer. :) And just like structural engineers are more careful and rigorous with the design for a hospital than they would be for a small home reno, software engineers are more careful with some projects (such as software intended for use in space exploration) than with others (a timer to remind you that your tea has finished steeping).
Masuri wrote:I would like to clone you and replace every development person in my company with your flesh copies, sir.

Masuri
Posts: 536
Joined: Sun Jul 22, 2007 8:23 pm UTC

Re: Computer science vs. software engineering?

Postby Masuri » Wed Nov 21, 2007 8:47 pm UTC

canuckotter wrote:Quick and dirty hacks in the right places are just as professional as a perfectly-engineered elegant solution, and over-engineered nonsense is just as unprofessional as ugly hacks in the wrong place.


I would like to clone you and replace every development person in my company with your flesh copies, sir.

canuckotter
Posts: 47
Joined: Mon Nov 12, 2007 4:27 pm UTC

Re: Computer science vs. software engineering?

Postby canuckotter » Wed Nov 21, 2007 9:10 pm UTC

:lol: Thanks! But actually, I still have a horrible tendency to overengineer things, partly because I spent so long being forced to underengineer things... I have to stop myself and think about everything I'm about to build to make sure I'm not making things more complicated than they need to be. I'm encouraged by the fact that I recently turned a three-month project into a three-week project just by clarifying the requirements. :lol:
Masuri wrote:I would like to clone you and replace every development person in my company with your flesh copies, sir.

User avatar
fortyseventeen
Ask for a lame title, receive a lame title
Posts: 88
Joined: Fri Mar 02, 2007 3:41 am UTC
Location: SLC, UT, USA
Contact:

Re: Computer science vs. software engineering?

Postby fortyseventeen » Thu Nov 22, 2007 12:55 am UTC

canuckotter wrote:
fortyseventeen wrote:EDIT: Note that I don't have anything against coding/hacking (not to be confused with agile development). It's a wonderful way to become familiar with or discover new concepts and frameworks, but trying to pass it off as professional/academic work is shameful.

Quick and dirty hacks in the right places are just as professional as a perfectly-engineered elegant solution, and over-engineered nonsense is just as unprofessional as ugly hacks in the wrong place.


Hey! What did I just say? I thought I told you not to confuse hacking with agile development! You didn't need to rant on for a page and a half on my tiny little comment.

I agree with the concept of everything you said, but it's the terminology that makes the difference. As far as I know, "hacking" or "just coding" refers to a type of mindset that is more or less indifferent to the fate of the project in question, i.e. as of one who fails to test their own solution. What you described was quick, but not necessarily dirty.

P.S. I especially agree with the comment about "over-engineered nonsense". I've had quite my share of that. :? Personally, I hope I can make it big enough on my own to rid myself of it...
Quick, what's schfifty-five minus schfourteen-teen?

canuckotter
Posts: 47
Joined: Mon Nov 12, 2007 4:27 pm UTC

Re: Computer science vs. software engineering?

Postby canuckotter » Thu Nov 22, 2007 1:52 am UTC

Oh trust me, I've seen some perfectly valid, completely professional solutions to problems that would be dirty in anyone's books. :lol: It's just that it doesn't matter sometimes. :)
Masuri wrote:I would like to clone you and replace every development person in my company with your flesh copies, sir.

Rysto
Posts: 1459
Joined: Wed Mar 21, 2007 4:07 am UTC

Re: Computer science vs. software engineering?

Postby Rysto » Thu Nov 22, 2007 3:15 am UTC

canuckotter wrote:Quick and dirty hacks in the right places are just as professional as a perfectly-engineered elegant solution, and over-engineered nonsense is just as unprofessional as ugly hacks in the wrong place.

I could not disagree more. I'll admit I don't have a lot of software development experience but I have had a variety of different experiences. And the one constant between all of those jobs, the one lesson I learned again and again in every single one of them was kludges tend to perpetuate themselves long past their useful life, and the longer a kludge has been in place, the longer and more painful it will be to get rid of it. I can think of plenty of times where I've cursed an awful hack, but I can't think of a single time where I've complained that a solution was over-engineered.

If it's worth doing, it's worth doing right.

Masuri
Posts: 536
Joined: Sun Jul 22, 2007 8:23 pm UTC

Re: Computer science vs. software engineering?

Postby Masuri » Thu Nov 22, 2007 3:53 am UTC

Rysto wrote:If it's worth doing, it's worth doing right.

You've inadvertently touched on the heart of the matter.

The problem here is that 'right' is subjective. I have to clear 80k obsolete records out of an application's database. Do I write a stored proc to do that? Or maybe a convoluted and parameterized perl script that takes a variety of inputs and might work for other situations like this one? Do I hop onto TOAD and delete the records? Do I truncate the table and reload it with the valid data? All of that stuff would work and are valid approaches - which one is doing it 'right?'

Yesterday I came across a chicken whose helicopter had malfunctioned for reasons unknown. The script was so esoteric that I had no idea what the guy was trying to accomplish, and since I was hired as his replacement when he quit, I have no way to ask him what the hell he was trying to do. I spent half the day untangling parameters and variables and strings and odd module calls to see that, yes, he wanted to get rid of obsolete records in the application database. So instead of his 130 line perl script which sourced 3 other 200-ish line parameter files, I used a 4 line shell script on the command line to call the utility in the application that is specifically for getting rid of obsolete records in a way that preserves the health of the app.

Who did it the right way? Technically, I did, if you look at the vendor's knowledge base. But his approach also had merit. He and I got to the same place, I just dispensed with the helicopter, picked up the chicken and carried it across the road.

User avatar
fortyseventeen
Ask for a lame title, receive a lame title
Posts: 88
Joined: Fri Mar 02, 2007 3:41 am UTC
Location: SLC, UT, USA
Contact:

Re: Computer science vs. software engineering?

Postby fortyseventeen » Thu Nov 22, 2007 4:13 am UTC

canuckotter wrote:Oh trust me, I've seen some perfectly valid, completely professional solutions to problems that would be dirty in anyone's books. :lol: It's just that it doesn't matter sometimes. :)


Quite true, which is why I carefully defined the term to include only cases in which it doesn't matter. I suppose it could still be considered professional, if you were in a pinch, and I also suppose I was referring to new projects, rather than ones that needed make-or-break maintenance.

Rysto wrote:If it's worth doing, it's worth doing right.


Yes, but additionally, from the standpoint of agile development, it can still be right even if it's quickly thrown together.

EDIT: ..if it's thrown together carefully.
Last edited by fortyseventeen on Thu Nov 22, 2007 4:26 am UTC, edited 3 times in total.
Quick, what's schfifty-five minus schfourteen-teen?

Rysto
Posts: 1459
Joined: Wed Mar 21, 2007 4:07 am UTC

Re: Computer science vs. software engineering?

Postby Rysto » Thu Nov 22, 2007 4:18 am UTC

In what world is a 730-line Perl script as good a solution as a 4-line shell script?

User avatar
fortyseventeen
Ask for a lame title, receive a lame title
Posts: 88
Joined: Fri Mar 02, 2007 3:41 am UTC
Location: SLC, UT, USA
Contact:

Re: Computer science vs. software engineering?

Postby fortyseventeen » Thu Nov 22, 2007 4:23 am UTC

Masuri wrote:Who did it the right way? Technically, I did, if you look at the vendor's knowledge base. But his approach also had merit. He and I got to the same place, I just dispensed with the helicopter, picked up the chicken and carried it across the road.


Heh. Technically, you just took the chicken out of his home-made heli with crooked wings, sputtering oil and put him in the one that was already built and tested, but yes. Good example.
Quick, what's schfifty-five minus schfourteen-teen?

Masuri
Posts: 536
Joined: Sun Jul 22, 2007 8:23 pm UTC

Re: Computer science vs. software engineering?

Postby Masuri » Thu Nov 22, 2007 4:32 am UTC

Rysto wrote:In what world is a 730-line Perl script as good a solution as a 4-line shell script?

Well, I think this is one of those guys who loves perl. I mean, loves it. Like other men love their wives. He plans for every possible situation that could ever happen. Aliens attack? Covered. Black hole swallows the Earth? No problem. Godzilla? Pshaw. You might want to port this to Windows even though the app is based only on UNIX? Done! My God, is there anything perl can't do?!

Problem is, he clearly had no idea how the app he was cleaning up actually works. His application-specific syntax was the only flaw in the helicopter. True, it was the only piece that was actually necessary, but I think the construction of the script was the goal, not the actual function.

(I have well and truly hi-jacked this thread, I think. >< )

User avatar
Hammer
Because all of you look like nails.
Posts: 5491
Joined: Thu May 03, 2007 7:32 pm UTC
Contact:

Re: Computer science vs. software engineering?

Postby Hammer » Thu Nov 22, 2007 4:41 am UTC

Masuri wrote:(I have well and truly hi-jacked this thread, I think. >< )

Yeah. I was just waiting for somebody to build the chicken a steam powered airplane. Why don't we continue this agile chicken discussion over in Religious Wars?
"What's wrong with you mathematicians? Cake is never a problem."

Rysto
Posts: 1459
Joined: Wed Mar 21, 2007 4:07 am UTC

Re: Agile Development or Bad Design? (AKA Chickens with Helicopt

Postby Rysto » Thu Nov 22, 2007 5:15 am UTC

Oh, don't even get me started an "Agile Development". Since time immemorial programmers have hacked away at programs without any kind of plan, and the result is inevitably an unmaintainable mess. But then somebody gave it a clever name and called it a "software development process", despite the fact that it's overriding feature is the lack of any processes whatsoever, and all of a sudden people are treating the idea like it has some kind of legitimacy. Never mind that it completely ignores 40 years of hard-won software engineering experience. Never mind that refactoring code is a painful process, and the pain is exponentially proportional to the amount of other code that interfaces with the refactored code. Never mind that any code that you throw away was a complete waste of your time, and intentionally setting out to waste your own time is an idiotic idea. It's "Agile". That sounds like a cool buzzword, so it must be a good idea!

canuckotter
Posts: 47
Joined: Mon Nov 12, 2007 4:27 pm UTC

Re: Computer science vs. software engineering?

Postby canuckotter » Thu Nov 22, 2007 7:23 pm UTC

Rysto wrote:I'll admit I don't have a lot of software development experience

I do. :D

Rysto wrote:I can think of plenty of times where I've cursed an awful hack, but I can't think of a single time where I've complained that a solution was over-engineered.

I have. This morning. And yesterday, for a separate issue. And last week. Surprisingly often, in fact, considering that we tend to lean towards the hack solution around here. :D

Rysto wrote:If it's worth doing, it's worth doing right.

Agreed. Where we differ is in our interpretation of "right". I'm guessing you're still in school... In school, and in early development work, you're almost always encouraged to overengineer the crap out of everything. If I ever manage to get a coop student in here to work with me, I'll be doing the same thing. The reason is that you need that experience of engineering everything in order to make most engineering tasks second nature, so you don't have to actually think about your design in order to come up with something effective. But after you've developed those instincts, the reality is that the "right" design is the one that is most effective in allowing the user to accomplish their goals.

Back to my example of the list that needs to be sorted... Sure, the ten-minute solution is kind of ugly. Let's even say that you find a bug with it and spend an hour hacking it back out and hacking in a different sort algorithm, and it's still kind of ugly. And then you find it's too slow and spend a whole day hacking the whole thing apart and rebuilding it again. My god, look at all the wasted time! ... only it's still less than the two weeks it would take to completely optimise and perfect a beautiful, elegant solution. And with the extra 8 1/2 working days you've saved, you can build a few extra features, making the overall software more useful. So maybe the ten-minute hack was ugly, and maybe the hour-long hack ended up being too inefficient, and the code's still a little ugly... but the software is still better than it would have been if you'd spent two weeks on it.

Knowing when you've got one of those situations versus a situation where an elegant, heavily-engineered solution is applicable... that's a skill you develop with practice. And since underengineering is more painful and damaging than overengineering, newbies are told to err on the side of overengineering. :D (Besides, it's not like most places actually expect any significant useful work out of people until they're at least two years out of school...)
Masuri wrote:I would like to clone you and replace every development person in my company with your flesh copies, sir.

Rysto
Posts: 1459
Joined: Wed Mar 21, 2007 4:07 am UTC

Re: Agile Development or Bad Design? (AKA Chickens with Helicopt

Postby Rysto » Thu Nov 22, 2007 8:44 pm UTC

Your sorting example is not a good one. In almost all cases your standard library's sorting algorithm is the way to go. It's easy for anybody else to understand, it requires way less work on your own part than coding your own sorting algorithm and it's way less bug-prone.

I am still in school, but I also have 2 years of software development experience. I've seen far too many cases where somebody made a small hack, and then somebody needed to extend the hack for a slightly different application, and then somebody else adds some more functionality, and the end result is an awful mess because the first guy didn't spend the time doing it right in the first place.

canuckotter
Posts: 47
Joined: Mon Nov 12, 2007 4:27 pm UTC

Re: Agile Development or Bad Design? (AKA Chickens with Helicopt

Postby canuckotter » Thu Nov 22, 2007 9:45 pm UTC

Rysto wrote:I am still in school, but I also have 2 years of software development experience. I've seen far too many cases where somebody made a small hack, and then somebody needed to extend the hack for a slightly different application, and then somebody else adds some more functionality, and the end result is an awful mess because the first guy didn't spend the time doing it right in the first place.

There are two problems there.

Problem A: Person one didn't realise that a lot of changes were probably going to be needed to that code. Maybe there's a good reason for that, maybe not. Not really a big problem, because for every A, there were another nine situations where A had equally good reason to believe that the code should be better engineered but chose not to do it, and it never mattered. As much as you might bitch about the ugliness of the code, the fact is that were it not for person three (see problem B) you'd have saved a TON of time by using the hack in 10 situations and only having to rewrite one of them.

Problem B: Person three didn't take the time to point out that hey, we're rewriting this code a lot, maybe we should redesign it instead of just hacking it. Here's where the real issue is. Hacking is perfectly acceptable and perfectly professional, but only if you stop hacking and start putting in the engineering effort where required. Most good developers I know have a three strike rule... The first time you have to update the hack, no problem. The second time, you pay more attention. But if you have to come back to it a third time, that's a sign that either the design is poor (even for a hack) or the code's going to be seeing a lot of use, and either way, it's time for a rewrite. So you tear out the hack code and put in high-quality, properly-designed code.

Of course, that depends on having great people. If you only have ordinary or even just good people, you're right in saying that tactic won't work. Having a great team means that a LOT of the rules change... The work gets a lot harder, but it's a hundred times more satisfying.

(And for comparison: Seven years software development experience, including being sole designer for two apps, one of which was a distributed enterprise app.)
Masuri wrote:I would like to clone you and replace every development person in my company with your flesh copies, sir.

User avatar
evilbeanfiend
Posts: 2650
Joined: Tue Mar 13, 2007 7:05 am UTC
Location: the old world

Re: Agile Development or Bad Design? (AKA Chickens with Helicopt

Postby evilbeanfiend » Fri Nov 23, 2007 5:43 pm UTC

Rysto wrote:Oh, don't even get me started an "Agile Development". Since time immemorial programmers have hacked away at programs without any kind of plan, and the result is inevitably an unmaintainable mess. But then somebody gave it a clever name and called it a "software development process", despite the fact that it's overriding feature is the lack of any processes whatsoever, and all of a sudden people are treating the idea like it has some kind of legitimacy. Never mind that it completely ignores 40 years of hard-won software engineering experience. Never mind that refactoring code is a painful process, and the pain is exponentially proportional to the amount of other code that interfaces with the refactored code. Never mind that any code that you throw away was a complete waste of your time, and intentionally setting out to waste your own time is an idiotic idea. It's "Agile". That sounds like a cool buzzword, so it must be a good idea!


wtf? agile has pretty tightly defined process (well depending on what you consider 'agile' e.g. XP), cowboy coders shooting scripts from the hip are not following an agile process, they are not following any process.

the refactoring problem is valid to a point though, there are some languages which don't play well with refactoring tools e.g. c and c++, this make agile harder in them.
in ur beanz makin u eveel

EvanED
Posts: 4330
Joined: Mon Aug 07, 2006 6:28 am UTC
Location: Madison, WI
Contact:

Re: Agile Development or Bad Design? (AKA Chickens with Helicopt

Postby EvanED » Mon Nov 26, 2007 5:56 am UTC

Today's Dilbert is strangely relevant:
Image

User avatar
rrwoods
Posts: 1509
Joined: Mon Sep 24, 2007 5:57 pm UTC
Location: US

Re: Agile Development or Bad Design? (AKA Chickens with Helicopt

Postby rrwoods » Mon Nov 26, 2007 3:58 pm UTC

I'd say Agile Development, like any other development process, is a great process in the right circumstances and when used properly. However, with Agile, it's much easier to fuck up that process than with other processes; hence the need for a smaller, more competent team than with other processes.

Test-Driven Development (which is a subset of Agile) is one of my favorites, in concept. They go a little far with some things, I think, but all in all it's a great concept and is a fine representation of what Agile Development can be (again, if used properly). The "Refactor" in Red/Green/Refactor might be unsettling -- refactoring can be "painful" -- but with the right tools (namely a code browser that can perform major refactors automatically), it isn't.
31/M/taken/US
age/gender/interest/country

Belial wrote:The sex card is tournament legal. And I am tapping it for, like, six mana.

btilly
Posts: 1877
Joined: Tue Nov 06, 2007 7:08 pm UTC

Re: Computer science vs. software engineering?

Postby btilly » Mon Nov 26, 2007 6:04 pm UTC

canuckotter wrote:Think about it this way... Imagine that as a tiny part of the overall program, you have a list that'll need to be sorted. You could spend a week building a super-fast optimised sort for that list, or you could spend ten minutes hacking in some existing code that's reasonably efficient but not great. Which would you use? If you chose either option, you're wrong.


Absolutely right. If in any normal language it takes you 10 minutes to use the standard built in library sort, then you are incompetent.

canuckotter wrote:A good software engineer would know that before he could answer the original question, he'd need to know how many items the list is likely to hold and how critical performance of the sort is going to be. If it's a short list (say, under a hundred items) and you spend more than ten minutes to hack in some existing reasonably-efficient sort, you're wasting time. Even if it's a long list, if the sorting's able to be done in the background (perhaps while also waiting for an HTTP request to return) then you should also probably use the 10-minute solution. The only time you should look at the week-long solution is if you're going to have a long list to sort and performance really is critical. (And even then, you should see if there's a way you can get away with not sorting the whole list all at once, or maybe even not sorting at all...)


Uh, wrong. Unless your program is being built to handle datasets large enough not to fit in memory, the standard library sort is probably just about ideal. The only real question most programmers should ever run into is whether to sort in your language or to use ORDER BY in the database.

If you're an exception to this rule, it should be obvious in your program's specification. (For instance I had to write some custom sorts to sort a dataset that the database was too small to sort, on a machine that was too small to hold the dataset in uncompressed form.)

canuckotter wrote:A good software engineer knows that there are times when careful design and forethought are required, and puts a lot of energy and attention into those situations when they come across them. The rest of the time, a good software engineer knows to just bang stuff together and make it happen and not worry about it too much, because for most stuff, "good enough" actually is. Knowing when to do a quick hack is just as important as knowing when not to, because the more time you spend engineering random crap, the less time you have for engineering the really important stuff.


Data point for you. The authors of Peopleware ran a series of instructive coding competitions. Not only did they verify the oft-repeated 100 to 1 productivity difference between programmers, but they studied the differences between the best and the average. What they found is that as you went up the productivity ladder the proportion of time spent coding consistently went down. Instead the best spent the most on other activities, like planning, testing, and reviewing code.

Which means that the best know better than to "just bang stuff together and make it happen". Instead engineering becomes an integral part of everything they do. But the engineering itself becomes invisible.

Rysto wrote:
canuckotter wrote:Quick and dirty hacks in the right places are just as professional as a perfectly-engineered elegant solution, and over-engineered nonsense is just as unprofessional as ugly hacks in the wrong place.

I could not disagree more. I'll admit I don't have a lot of software development experience but I have had a variety of different experiences. And the one constant between all of those jobs, the one lesson I learned again and again in every single one of them was kludges tend to perpetuate themselves long past their useful life, and the longer a kludge has been in place, the longer and more painful it will be to get rid of it. I can think of plenty of times where I've cursed an awful hack, but I can't think of a single time where I've complained that a solution was over-engineered.

If it's worth doing, it's worth doing right.


You're right that kluges have a long life expectancy. In fact for most industries about 80% of the final cost comes during maintenance, and that is where badly thought out hacks come back to bite you. So don't do that.

However practice moderation. Some of the worst code out there is overengineered. People put a lot of energy into making their code as general and flexible as possible. But they did so without good insight into the ways that said code would actually be modified. The result is that the intent of the code is hideously obscured, and to make the change that you need to make you need to fight the way they engineered it.

My rule of thumb is, "Don't try to abstract a problem until you're solving it for the third time." You need the previous experiences to get a good idea what design problem you need to solve for. Until then you should just build the clearest thing you can which does what is needed.

Which is never a bad idea. Good designs tend to have a deceptive simplicity. They look straightforward and obvious. It is only after you've used the code for a bit that you realize, "Hey, working with this is easy. How was it done?" Then you start to appreciate the thought that must have gone into the design.
Some of us exist to find out what can and can't be done.

Others exist to hold the beer.

User avatar
davean
Site Ninja
Posts: 2498
Joined: Sat Apr 08, 2006 7:50 am UTC
Contact:

Re: Computer science vs. software engineering?

Postby davean » Tue Nov 27, 2007 1:59 pm UTC

btilly wrote:Uh, wrong. Unless your program is being built to handle datasets large enough not to fit in memory, the standard library sort is probably just about ideal. The only real question most programmers should ever run into is whether to sort in your language or to use ORDER BY in the database.


It is just about ideal *unless* you know something more about the data then the ability to determine if one key is less then another (which is what standard library sorts are built around) and you need it to be fast. Props to your language if it has a sort that can use relative positions (Like Radix), but thats just one type of extra information. There are plenty of times when you know a decent bit more about the data then ether a basic comparison (the definition of sorted) or it's prefix (and your language/libraries probably can't use that by default).

In a performance constrained environment, I'm going to have to disagree with you. Most places just throw money at it though. Often other people's money.

btilly
Posts: 1877
Joined: Tue Nov 06, 2007 7:08 pm UTC

Re: Computer science vs. software engineering?

Postby btilly » Tue Nov 27, 2007 5:51 pm UTC

davean wrote:
btilly wrote:Uh, wrong. Unless your program is being built to handle datasets large enough not to fit in memory, the standard library sort is probably just about ideal. The only real question most programmers should ever run into is whether to sort in your language or to use ORDER BY in the database.


It is just about ideal *unless* you know something more about the data then the ability to determine if one key is less then another (which is what standard library sorts are built around) and you need it to be fast. Props to your language if it has a sort that can use relative positions (Like Radix), but thats just one type of extra information. There are plenty of times when you know a decent bit more about the data then ether a basic comparison (the definition of sorted) or it's prefix (and your language/libraries probably can't use that by default).

In a performance constrained environment, I'm going to have to disagree with you. Most places just throw money at it though. Often other people's money.


Even if you're in a performance constrained environment, for a first pass you usually still should just use the library sort. Then profile and revisit it later.

See http://stevemcconnell.com/cctune.htm for why I say that.
Some of us exist to find out what can and can't be done.

Others exist to hold the beer.

canuckotter
Posts: 47
Joined: Mon Nov 12, 2007 4:27 pm UTC

Re: Agile Development or Bad Design? (AKA Chickens with Helicopt

Postby canuckotter » Tue Nov 27, 2007 5:54 pm UTC

The sort thing was just the first example that popped into my head of where someone could waste a lot of time overengineering stuff -- or underengineering stuff. And your "use the standard sort" isn't a great answer anyway, because the specific situation I was thinking of, the program's retrieving 100K+ objects from multiple sources in arbitrary order and needs to display them... Why sort the list when you could just grab the top N (N = number of items that display on the screen) first and then sort the rest once the UI has been populated? It's not going to be pretty code, unless you spend a lot of time designing a higher-level sort that'll give you that functionality in a nicely wrapped package... but why bother? It's not like this is a problem that comes up a lot around here, it's only a very specific situation where we need to worry about getting that much data at once, so why put the effort into building an abstracted solution that'll never be used again? Just build the ugly solution and move on.

Of course, if it does come up again, then we'd look at building a smarter solution.

(And before anyone mocks me: yes, there are better solutions than the one I posted. For example, as the objects come in, you could check to see if they're within the top N so far, and if so, put them in place... otherwise, toss them at the end of the list and only sort the rest when you've got the full list downloaded from all sources.)

As for "agile development"... I subscribe to the school of thought that says that the better your developers, the worse of a fit Methodologies become. :lol: Good development teams will come up with their own ways of making things happen as efficiently as possible, and things that are perfect for one team could be nothing but pain for another. The "Three strikes and rewrite" rule is a pretty common one among great developers, from what I've heard. If it also happens to exist within "agile", well, great. I don't care one way or the other. ;)
Masuri wrote:I would like to clone you and replace every development person in my company with your flesh copies, sir.

btilly
Posts: 1877
Joined: Tue Nov 06, 2007 7:08 pm UTC

Re: Agile Development or Bad Design? (AKA Chickens with Helicopt

Postby btilly » Tue Nov 27, 2007 6:32 pm UTC

canuckotter wrote:The sort thing was just the first example that popped into my head of where someone could waste a lot of time overengineering stuff -- or underengineering stuff. And your "use the standard sort" isn't a great answer anyway, because the specific situation I was thinking of, the program's retrieving 100K+ objects from multiple sources in arbitrary order and needs to display them... Why sort the list when you could just grab the top N (N = number of items that display on the screen) first and then sort the rest once the UI has been populated? It's not going to be pretty code, unless you spend a lot of time designing a higher-level sort that'll give you that functionality in a nicely wrapped package... but why bother? It's not like this is a problem that comes up a lot around here, it's only a very specific situation where we need to worry about getting that much data at once, so why put the effort into building an abstracted solution that'll never be used again? Just build the ugly solution and move on.


A standard sort will take less than 2 million comparisons. Getting the top 20 will still take 100,000 comparisons. Plus whatever else you have to do to extract data. Much of which presumably will involve remote I/O. In which case there is a good chance that the performance with the sort will be relatively minor compared to the other work you have to do.

canuckotter wrote:Of course, if it does come up again, then we'd look at building a smarter solution.


That would be a heap sort. :D
Some of us exist to find out what can and can't be done.

Others exist to hold the beer.


Return to “Religious Wars”

Who is online

Users browsing this forum: No registered users and 8 guests