Estimating software project deadlines
Let me tell you. It’s hard.
There are managers who never learn why software project deadlines sometimes can’t be estimated with accuracy. There are programmers who never learn how to estimate deadlines with any kind of accuracy. How do we deal with this?
Having an accurate project deadline means schedules can be planned. It means the future can be glimpsed and foretold, albeit somewhat uncertainly. It means we are in control.
Real life doesn’t happen this way. Real world applications are infused with innovation, or they should be for any kind of improvement. This means new concepts, new ways of interacting, new ways of thinking about data, new ways of improving existing processes. New, new, new.
I’ve worked on both long-term huge projects and small-term mini projects. I’ve learned that there are many variables affecting project deadline estimation and the actual project completion date. Some of them are
- skill of individual programmer
- understanding of the project/business logic
- affinity with other programmers on the team
- difficulty of the task itself
I’m continually given small projects and asked to give an estimate on how long I’d take to complete them. My experience is that the smaller the task, the easier it is to estimate. The more standard the task, the easier it is to estimate.
This doesn’t mean you can’t innovate. On the contrary, it means you must innovate even more. It means breaking down any project into bite-sized portions that you can estimate with certainty. It means you must understand the project on many levels, from how the project fits in with existing projects to how a single line of code fits in.
And it’s hard. Understanding on so many levels require a lot of thinking. It takes some serious brain mojo to figure out that on the macro scale, data is entered and stored in a certain way, and that on the micro scale, a particular if condition has to be coded, even though the condition is never mentioned, implicitly or explicitly.
It involves asking a lot of what if’s. What if the user keyed in this value? What if the user keyed in this combination of values?
The naive manager never understands how a simple project with simple interfaces require so much time, because hidden within the project are assumptions that manager thinks are obvious. The unskilled programmer never learned to gauge his work, because there are assumptions hidden within the project and need to be explicitly coded.
When I’m asked for an estimate, what they’re really looking for is, “How many man days are required?” So far, my answers range anywhere from 5 to 35 man days. In case you’re wondering, 5 man days means 5 work days. It can be 5 people cooperating to finish the task in 1 day, or 1 person doing the task in 5 days. This method gives a projection of the deadline and the cost involved (in paying the programmers).
So how can you improve the accuracy of your estimates? Two factors, skill and standards. Your thinking and programming skills make it easy to translate business logic into code. Which gives you greater confidence in estimating. The more standard a task is visualised, the more previous knowledge can be leveraged. Which reduces new concepts and code, and give you greater confidence in estimating.
Ultimately, it is simply a matter of reducing errors and uncertainty in project estimation. I wonder if L-1 linear regression or least square approximation can be used for projecting future deadlines? I’d have to go through all my documentation to find out what my estimate was and the actual time taken was for any given project… It will require enough data points to sufficiently draw a line.
On that note, FogBugz uses Evidence-Based Scheduling, involving up to 200 data points. Instead of giving you when a project can complete, it gives you a probability of likelihood that the project can complete on a certain date. For example, there’s a 50% likelihood that your project can complete by next week. But if you can extend it another week, there’s a 80% likelihood that the project can complete.
I neither have the time and effort to do least square approximations, nor the software to do historic statistical analysis of previous estimations for future estimations. I do have my experience and programming skill, so basically I trust my gut.
So, how do you estimate your software project deadlines?
Upgrading WordPress 2.2 to 2.3
So WordPress version 2.3 was released on 24th September 2007. The question is, do you upgrade?
I’m just getting comfortable with WordPress, and then they announce a major upgrade release. As I see it, there are 4 types of people: bleeding-edgers, early adopters, late adopters and the non-budgers. I usually hover between the first two groups. In the case of WordPress, my procrastination could be summed up in one word: laziness.
I’ve been thinking of upgrading for a while already, so finally, one sunny Sunday morning, I went through the whole upgrading process. Here’s what I did (following instructions from the WordPress upgrade page):
1. Backup my database
This one’s easy. The backup instructions provided by WordPress are easy to follow, and there are pictures. Can’t see how anyone can fumble on this.
2. Backup my files
WordPress suggests that I backup all the WordPress files. I thought that since I’m backing up stuff, I might as well go the whole nine yards and backup everything. So I downloaded my entire site onto my computer.
Luckily I’ve got SmartFTP to help me out. I set the download in progress and went for tea. Seriously. The download is gonna take a while…
I also did an export of the posts from within the WordPress panel into the .xml file. Just in case.
3. Verify the backups
Ok, verified the SQL statements are correctly formed in the backup file. I’ve got the entire site on my computer. Moving on…
4. Downloaded all the upgraded plugins
Starting to veer away from the instructions given by WordPress. My reasoning is that I want the upgrade process to happen so fast that even Superman won’t notice any site downtime. So I want everything that can be done without involving the site to be done. This include any downloading of plugin updates.
I’ve got these plugins:
- FeedBurner FeedSmith
- Dagon Design’s Sitemap Generator
- Google Sitemap Generator
- SEO Title Tags
- Ultimate Tag Warrior
- Mark Jaquith’s Subscribe to Comments
- Douglas Karr’s Contact Form
- Text-Link-Ads
I was able to find updates for all the plugins except Ultimate Tag Warrior, SEO Title Tag and Subscribe to Comments. Well, version 2.3 has in-built tagging capabilities, and I found UTW to be a bit intimidating when I set it up. I don’t need that much flexibility anyway, so UTW is dropped out. It was one of the major reasons why I wanted to upgrade in the first place, for the native tagging capability.
As of this writing, Netconcepts announced a working version of SEO Title Tag for WordPress 2.3 by 31 October. Well, I’m already in the mood for upgrading, so I’ll upgrade later. From what I found, there’s only minimal incompatibilities. Official word from WordPress compatibility results shows that “Primary function is fine but returns a fatal error when tag pages are accessed.” Hmm… oh whatever, so I just went ahead and installed what Netconcepts had first.
As for Subscribe to Comments… no news from Mark, and I made a decision and dropped the plugin. It hasn’t been of frequent use, so it’s not too bad.
5. Downloaded and extracted the WordPress 2.3 package
Same reasoning as above. I mean, what if my Internet connection went limp? You mean I deactivated my plugins, and my site’s sitting there incomplete, and I’m here waiting for my download to complete? No, no, no…
So I downloaded the package first, and extracted everything.
Got a bonus. The stable release is version 2.3.1, so I got something extra. Hope 2.3.2 is a bit further in the future…
6. Deactivate plugins
Now I deactivate the plugins. I was going down the line and when I reached the bottom, I noticed there’s a “Deactivate All Plugins” link provided. Stupid me. Oh well, I didn’t have a lot of plugins anyway…
7. Ensure necessary steps are done
I didn’t follow the WordPress instructions as given. But the idea was to make sure database backup was done (and usable), file backup was done (and usable) and plugins deactivated. Checked.
8. Delete old WordPress files
This one I followed to the letter. I don’t mess around with stuff I don’t fully know or understand. I did make one mistake. I deleted the wp-config.php file. Aahhh! I didn’t realise it until I couldn’t do the upgrade process. More on this later…
9. Upload the new WordPress files
SmartFTP saved the day again. A few clicks and everything’s on their merry way. I went to get another cup of tea…
10. Run the WordPress upgrade program
Basically, you fire up your favourite browser, and type http://example.com/wp-admin/upgrade.php into the address bar (assuming example.com is your site). Or http://example.com/blog/wp-admin/upgrade.php if you installed on a subdirectory “blog” when you first installed WordPress.
This was the point where I realised I couldn’t upgrade because I was missing the wp-config.php file, which I was supposed to leave alone. Luckily, I made a full backup. See how useful that was? Whew… I uploaded the wp-config.php from my backup, and ran the upgrade program again. It worked like a charm!
11. Update Permalinks and .htaccess
Nothing to do really. But I clicked the update button anyway.
12. Installed updated plugins
I uploaded all the updated plugin .php files, and reactivated them. Except for Ultimate Tag Warrior. I went through all the plugins and saved the settings. Particularly the Google Sitemap Generator, since I’ve got to rebuild the sitemap.
13. Checked that site is working
Failed. Because I was using an Ultimate Tag Warrior function for displaying tags. Ok, deleting from my .php files. Site restored, sans tags.
14. Import Ultimate Tag Warrior tags to WordPress
This turned out to be easier than I thought. Look at Technical Itch and Rich’s two part article for more information. Simply go to the WordPress control panel, select [Manage], then [Import], then scroll down to find the Ultimate Tag Warrior link.
Click to enlarge screenshot.
There are 3 steps to the whole thing, although it’s stated there are 5. Maybe it depends on the blog structure. No radio buttons. No check boxes. No typing. Just keep clicking on the [Next] button…
Now to display the tags. Use this format
<?php the_tags('before', 'separator', 'after'); ?>
Cryptic. What before represents is the text before the tags, separator represents the text between tags, and after represents text after the chunk of tags. An example explains this better:
<?php the_tags('Internal tags : ', ' | ', ''); ?>
Which gives something like
Internal tags : polymath | code | upgrade
Theoretically (I haven’t tried this), you can set it to be
Internal tags: <?php the_tags('<ul><li>', '</li><li>', '</li></ul>'); ?>
to get
Internal tags: <ul><li>polymath</li><li>code</li><li>upgrade</li></ul>
This gives you some flexibility with CSS for a more customised display.
Anyway, that chunk of php code was added into my index.php and single.php, just after the post content. Uploaded the two files and rechecked blog. Yay! We’ve got tags!
One small note: The imported tags have a dash in between words. So if the original UTW tag is “original tag”, it’s displayed as “original-tag”. The original UTW tags are stored with a dash anyway (you should have noticed it when you save your drafts with tags).
It’s not a big deal. Perhaps when I’m in the mood, I might go into the database and flex my SQL powers to do update statements. Right now, I’m just glad everything went smoothly.
15. Write a post
This one’s not in the official upgrade guide. What good is an upgrade if you can’t use it, right? So I went ahead to write up a new post. You’re reading it!
I came across a problem I’ve never come across before. I can’t save my post. What!? After all that hard work, I can’t save a post? Then I checked my post length, which isn’t much longer than usual. I deleted some text and tried. It went through.
Hmm… I added some text and it failed with a 404. The threshold seems to be about 4000 characters, but I don’t think that’s the problem. I did some searching and found this solution about saving with an “Error 404″. Simply add
SecFilterEngine Off SecFilterScanPOST Off
to your .htaccess file (located at the root of your blog). Problem solved.
[Update] If you still have a problem, try placing the two lines within an IfModule tag, like so
<IfModule mod_security.c> SecFilterEngine Off SecFilterScanPOST Off </IfModule>
Congratulations!
I have successfully upgraded my blog from WordPress 2.2 to 2.3. The entire process took about 3 hours, with the better part of it used on downloading and uploading stuff. Actual manual interaction was much less. I drank tea and dozed lightly in front of the computer most of the time, keeping tabs on the whole process.
I’m so proud of myself. Since it’s hard to pat myself on my back with my hands, I’m going to go get myself a cup of tea instead…
Tell me about your experience with upgrading in your comments!
Are you helping yourself out of a job?
So I like helping people. In university, during assignment periods when the professor hands out programming tasks, I’m usually one of the first few (if not the first) to complete the programs. My fellow classmates would then pull me (sometimes literally) to their workstations and ask me to debug their programs. After many of these episodes (and the fading feeling of everyone wanting a piece of me), I learned to restrain myself, giving hints and partial answers instead. It’s their grades, and I’m not taking their exams, because I’ve got my own. I felt it’d benefit them more.
Moving on to working life, I found myself, strangely, in a similar position. Not on quite as big a scale as in university though. In one company, there was this guy who really didn’t quite cut it as a programmer. He’s still hired to help the team, so all of us gave our best shot at helping him and easing him into the team.
I remember staying back (together with another colleague) with him till quite late, helping him work through an issue. Alright, I hated it. It was clearly his problem, and there were moments when I really wanted to take over his computer to try out solutions and strangle him, not necessarily in that order.
In another company, I met another programmer who needed help with her work too. The problems were fairly simple, involving the improper use of for loops, if conditions as well as some design logic mistakes. During one of our conversations, I realised something. She was related to the programmer mentioned from before! Surprise, surprise… I kept my discovery to myself and slowly but surely, removed myself from her social radar…
Meeting these people, and helping them, I asked myself, “Am I helping myself out of a job?” Since I’ve started participating in the Dream In Code forums, I’ve had a feeling of being needed again. Recent work life drained a lot out of me, and it felt good to be able to help people with programming problems. Then someone pointed out if newbies will kill the programming star, because we might inadvertently create incompetent programmers who just seem to be competent.
The ideas presented were relevant, and since I’m in the programming profession, I could understand the feelings of frustration and despair. What used to be their problem, became your problem. And after it’s solved, it became their work, while you got nothing.
This actually reminded me of something my professor said (from a few years back). My professor and I were discussing my thesis (computer viral epidemiology), and he said that there were cases where the mentor took credit for a paper his student wrote. Here comes the dilemma for the student. A student, being the unrecognised nobody that he was, needs the mentor (and the mentor’s reputation) to break into the academic circle. How does one claim credit for a paper, if one does not tell anyone? To claim credit, he’d have to tell the mentor and risk his work being stolen. If he doesn’t tell the mentor, he can’t claim credit. Either way, the student loses. Unless the mentor is honest of course.
This is similar to the prisoner’s dilemma problem. Two criminals working together are caught and both are subjected to interrogation separate from each other. Being the idiot criminals that they are, both try to betray the other (and hopefully gain some benefits and leeway), so the good guys get all the evidence needed to put both of them down. If one keeps quiet (or keep to their end of the story), but the other divulge all, then the betrayer gets to go free (or get minimum punishment), while the other gets jail time big time.
If you’re going to be punished anyway, then there’s no reason to keep quiet, is there? Just defect, since the outcome is bad anyway. Well, there’s actually one more outcome, and that’s if both of them cooperate and keep quiet (or keep to the story). Both gets punished, but not as much as if one got betrayed. And I’m not condoning criminal activity here…
Where am I going with all this? If you help someone, your fellow colleague or student or any potential perceived competitor, are you setting yourself to be out of a job?
I just finished watching an episode of “Numb3rs”, where Don, the FBI agent, was visiting a psychiatrist to deal with some personal issues. The topic of leading his team came up, and towards the end of the show, he realised that he’s afraid of letting his team go, of letting his team stand up for themselves. Of his team not needing him anymore. Then he understood that when his team excels, he excels.
Ultimately, I believe helping those fellow programmers improve their skills is beneficial. It depends on the state of mind and belief of the (better) programmer. Cooperate with them more than competing with them. Help one another to make better software. The world needs better software.
And those unscrupulous programmers who are out to get credit yet do little on their own? Fade them slowly out of your life, and concentrate on the people who do appreciate the help you give. Did you know teaching others programming actually make you a better programmer? Because in order to teach something, you have to be great at it.
Or do you think you’re not up to the challenge?
Post your comments! I’d love to hear your side of the story.
Reverse Polish Notation with C#
What is reverse polish notation? It means rewriting a mathematical expression into a postfix notation form. For example, “1 + 5 * 2″ is rewritten into “1 5 2 * +”.
Why would anyone want to do that? I actually wrote a C# tutorial on this at Dream In Code, with my explanations there. I wanted to make it really comprehensive, so I edited and reedited my draft. The draft turned out to be more than 4000 words, and that’s not even including the code segments.
Well, apparently the forum engine couldn’t handle my long article, so it’s been shortened. And I really worked hard at the article too…
Anyway, someone at the forum asked something about the Mandelbrot fractal and how to generate colours to make it “pretty”. I gave my take on the matter, but somehow I felt more could be said. It reminded me of a time back in the past …
I was young and full of drive and found the music visualisation in Windows Media Player mesmerising. So I researched on how I could make my own music visualisation software. I needed it to take on different patterns, and I didn’t want to hardcode stuff in.
It was at this point where I stumbled upon reverse polish notation (RPN). RPN allows me to form mathematical expressions and evaluate them in the program code. I was working with C then, so I had for loops and lots of forward and backward checks going through every single character when reading in the expression.
It was terrible. When I got an “s”, I had to check if the next two characters were “i” and “n” respectively, then I could be sure it’s a sine function. I couldn’t find my C code anymore, but I remember that I got it working.
Basically, tokenising the expression was the hard part. Once the tokens were obtained, parsing the expression and evaluating it was easy.
So, back to my tutorial. That question about Mandelbrot fractals got me thinking, “Maybe I can write something on RPN?” With C#, I could use regular expressions to recognise all the different tokens available. So I did that. The thing was, explaining RPN wasn’t the hard part. Explaining the queue and stack data structures, and how regular expressions worked, is. And I used the better part of my tutorial explaining that. We’re going for comprehensive here.
There are two things I left out in the tutorial. One is the use of variables in the expression. The other is adding more functions. Right now only sine, cosine and tangent are supported. The regular expressions used won’t be able to handle hyperbolic sine “sinh” for example. It would fail to recognise whether a token was “sin” or “sinh”. Oh well, maybe another tutorial…
The complete code can be downloaded at the tutorial page.
Blog Action Day 2007 - Use less paper
It’s Blog Action Day, and the topic for this year is the environment. It’s a movement where bloggers around the world blogs about one topic, on one day.
So I was thinking to myself, “What can I talk about, relating to the environment, coming from a programmer’s point?”
I struggled a bit. I’ve thought of recycling bottles and cans. Never really put much effort into it, although I buy bottled or canned drinks infrequently. Using paper bags more often then plastic ones? Yeah, and not nearly enough to quite justify it. Paper? Hmm… now that’s something. I’m accustomed to scribbling down notes on something like an envelope or printing paper or notepad. Then I continue scribbling new notes on those somethings until I can’t find another inch of whitespace.
When I first started working, I was actually appalled at the amount of paper used every month. I was learning the ropes at maintaining an application, and every month, I was to print out several reports and tally the results with each other. After checking that they were correct, I would then file them, and they’d be stuck in some office room corner where they’d never see the light again. Unless a disaster came up, such as a fire or earthquake or some user complaining about incorrect figures.
Then there’s the code. For some reason, the programmers of yore coded with such brutality and inefficiency and unreadability, that sometimes I had to print the code out. I couldn’t make head or tail of what they were trying to accomplish. So I thought holding the code in my hands would give me new insights. It was unbelievably stupid of me. It is in the nature of program code to have lots of whitespace. Sometimes, a curly bracket is all that lies on a single line.
So I tried to be clever, and printed the code as two pages on one side of the paper (you know, the landscape orientation). It was another unfathomably stupid mistake. I reduced the font to size 8 points on Courier New, and the resulting printout was barely visible. I had to squint to make out printf() statements. Absolutely terrible.
I also had the opportunity to attend meetings. Arriving at the meeting room, I’d only have my notebook (the paper kind). Other people would be carrying files of papers and the entire email printout of the meeting notes. And let me tell you, some of the email discussions were very lengthy, and with Outlook doing automatic indents for new replies, text were word-wrapped like crazy. A meeting with ten people probably used enough paper to kill a young sapling.
Another observation I made was the reliance of “hardcopies” of some users. They feel unsafe if there were no printouts of whatever it was they wanted, be it documentation or reports. Reports were printed out even if no one would ever read them. It was the monthly routine, and someone continued to print them out and file them neatly in a nice ring folder. It’s a waste of program resources, human effort and paper.
I understand it if you need to print something out. I’ve printed documents before, like program specifications. It’s just easier to have the business logic printed on paper in your hands, then code on the computer. Alt-Tabbing between a Word document and the Visual Studio is tough, even if I am getting better at it.
What can you do? Simply print less. You don’t have to forgo printing all together. A page saved here, a page saved there, and a tree could have lived a little longer. I’m sure there are parts of the document that you’ll never ever need. So print out the parts that you do need. Sometimes, program specs change, and then what are you going to do? Print it all out again?
Recently, I had the occasion to organise my desk into some semblance of neatness, and I finally decided to do something about a pile of files with documents that were like 10 years old (not mine though). I very nearly took them to the shredding machine and then I stopped. I flipped the document pages over, and they were blank on the other side. I suddenly had an enormous pile of rough paper for scribbling notes! Now to get all the staples out of the way…
I think that ultimately, going all out to save the environment may not be the answer, nor even necessary. Just use less. Use less bottles and cans and paper. Starting small means it’s easier to actually start.
