31 October, 2007 | Written by Vincent 4 Comments

Estimating software project deadlines

Target date Let me tell you. It’s hard.

There are managers who never learn why software project deadlines sometimes can’t be estimated with accuracy. There are programmers who never learn how to estimate deadlines with any kind of accuracy. How do we deal with this?

Having an accurate project deadline means schedules can be planned. It means the future can be glimpsed and foretold, albeit somewhat uncertainly. It means we are in control.

Real life doesn’t happen this way. Real world applications are infused with innovation, or they should be for any kind of improvement. This means new concepts, new ways of interacting, new ways of thinking about data, new ways of improving existing processes. New, new, new.

I’ve worked on both long-term huge projects and small-term mini projects. I’ve learned that there are many variables affecting project deadline estimation and the actual project completion date. Some of them are

skill of individual programmer
understanding of the project/business logic
affinity with other programmers on the team
difficulty of the task itself

I’m continually given small projects and asked to give an estimate on how long I’d take to complete them. My experience is that the smaller the task, the easier it is to estimate. The more standard the task, the easier it is to estimate.

This doesn’t mean you can’t innovate. On the contrary, it means you must innovate even more. It means breaking down any project into bite-sized portions that you can estimate with certainty. It means you must understand the project on many levels, from how the project fits in with existing projects to how a single line of code fits in.

And it’s hard. Understanding on so many levels require a lot of thinking. It takes some serious brain mojo to figure out that on the macro scale, data is entered and stored in a certain way, and that on the micro scale, a particular if condition has to be coded, even though the condition is never mentioned, implicitly or explicitly.

It involves asking a lot of what if’s. What if the user keyed in this value? What if the user keyed in this combination of values?

The naive manager never understands how a simple project with simple interfaces require so much time, because hidden within the project are assumptions that manager thinks are obvious. The unskilled programmer never learned to gauge his work, because there are assumptions hidden within the project and need to be explicitly coded.

When I’m asked for an estimate, what they’re really looking for is, “How many man days are required?” So far, my answers range anywhere from 5 to 35 man days. In case you’re wondering, 5 man days means 5 work days. It can be 5 people cooperating to finish the task in 1 day, or 1 person doing the task in 5 days. This method gives a projection of the deadline and the cost involved (in paying the programmers).

So how can you improve the accuracy of your estimates? Two factors, skill and standards. Your thinking and programming skills make it easy to translate business logic into code. Which gives you greater confidence in estimating. The more standard a task is visualised, the more previous knowledge can be leveraged. Which reduces new concepts and code, and give you greater confidence in estimating.

Ultimately, it is simply a matter of reducing errors and uncertainty in project estimation. I wonder if L-1 linear regression or least square approximation can be used for projecting future deadlines? I’d have to go through all my documentation to find out what my estimate was and the actual time taken was for any given project… It will require enough data points to sufficiently draw a line.

On that note, FogBugz uses Evidence-Based Scheduling, involving up to 200 data points. Instead of giving you when a project can complete, it gives you a probability of likelihood that the project can complete on a certain date. For example, there’s a 50% likelihood that your project can complete by next week. But if you can extend it another week, there’s a 80% likelihood that the project can complete.

I neither have the time and effort to do least square approximations, nor the software to do historic statistical analysis of previous estimations for future estimations. I do have my experience and programming skill, so basically I trust my gut.

So, how do you estimate your software project deadlines?

29 October, 2007 | Written by Vincent 5 Comments

Upgrading WordPress 2.2 to 2.3

UPDATE: The latest versions of WordPress (as of 12 July 2009) all have upgrading options in the administration panel. So you only need to backup your database, click that upgrade link/button in the admin panel, and that’s it. Just go get the latest version, and save yourself the hassle.

So WordPress version 2.3 was released on 24th September 2007. The question is, do you upgrade?

I’m just getting comfortable with WordPress, and then they announce a major upgrade release. As I see it, there are 4 types of people: bleeding-edgers, early adopters, late adopters and the non-budgers. I usually hover between the first two groups. In the case of WordPress, my procrastination could be summed up in one word: laziness.

I’ve been thinking of upgrading for a while already, so finally, one sunny Sunday morning, I went through the whole upgrading process. Here’s what I did (following instructions from the WordPress upgrade page):

1. Backup my database

This one’s easy. The backup instructions provided by WordPress are easy to follow, and there are pictures. Can’t see how anyone can fumble on this.

2. Backup my files

WordPress suggests that I backup all the WordPress files. I thought that since I’m backing up stuff, I might as well go the whole nine yards and backup everything. So I downloaded my entire site onto my computer.

Luckily I’ve got SmartFTP to help me out. I set the download in progress and went for tea. Seriously. The download is gonna take a while…

I also did an export of the posts from within the WordPress panel into the .xml file. Just in case.

3. Verify the backups

Ok, verified the SQL statements are correctly formed in the backup file. I’ve got the entire site on my computer. Moving on…

4. Downloaded all the upgraded plugins

Starting to veer away from the instructions given by WordPress. My reasoning is that I want the upgrade process to happen so fast that even Superman won’t notice any site downtime. So I want everything that can be done without involving the site to be done. This include any downloading of plugin updates.

I’ve got these plugins:

I was able to find updates for all the plugins except Ultimate Tag Warrior, SEO Title Tag and Subscribe to Comments. Well, version 2.3 has in-built tagging capabilities, and I found UTW to be a bit intimidating when I set it up. I don’t need that much flexibility anyway, so UTW is dropped out. It was one of the major reasons why I wanted to upgrade in the first place, for the native tagging capability.

As of this writing, Netconcepts announced a working version of SEO Title Tag for WordPress 2.3 by 31 October. Well, I’m already in the mood for upgrading, so I’ll upgrade later. From what I found, there’s only minimal incompatibilities. Official word from WordPress compatibility results shows that “Primary function is fine but returns a fatal error when tag pages are accessed.” Hmm… oh whatever, so I just went ahead and installed what Netconcepts had first.

As for Subscribe to Comments… no news from Mark, and I made a decision and dropped the plugin. It hasn’t been of frequent use, so it’s not too bad.

5. Downloaded and extracted the WordPress 2.3 package

Same reasoning as above. I mean, what if my Internet connection went limp? You mean I deactivated my plugins, and my site’s sitting there incomplete, and I’m here waiting for my download to complete? No, no, no…

So I downloaded the package first, and extracted everything.

Got a bonus. The stable release is version 2.3.1, so I got something extra. Hope 2.3.2 is a bit further in the future…

6. Deactivate plugins

Now I deactivate the plugins. I was going down the line and when I reached the bottom, I noticed there’s a “Deactivate All Plugins” link provided. Stupid me. Oh well, I didn’t have a lot of plugins anyway…

7. Ensure necessary steps are done

I didn’t follow the WordPress instructions as given. But the idea was to make sure database backup was done (and usable), file backup was done (and usable) and plugins deactivated. Checked.

8. Delete old WordPress files

This one I followed to the letter. I don’t mess around with stuff I don’t fully know or understand. I did make one mistake. I deleted the wp-config.php file. Aahhh! I didn’t realise it until I couldn’t do the upgrade process. More on this later…

9. Upload the new WordPress files

SmartFTP saved the day again. A few clicks and everything’s on their merry way. I went to get another cup of tea…

10. Run the WordPress upgrade program

Basically, you fire up your favourite browser, and type http://example.com/wp-admin/upgrade.php into the address bar (assuming example.com is your site). Or http://example.com/blog/wp-admin/upgrade.php if you installed on a subdirectory “blog” when you first installed WordPress.

This was the point where I realised I couldn’t upgrade because I was missing the wp-config.php file, which I was supposed to leave alone. Luckily, I made a full backup. See how useful that was? Whew… I uploaded the wp-config.php from my backup, and ran the upgrade program again. It worked like a charm!

11. Update Permalinks and .htaccess

Nothing to do really. But I clicked the update button anyway.

12. Installed updated plugins

I uploaded all the updated plugin .php files, and reactivated them. Except for Ultimate Tag Warrior. I went through all the plugins and saved the settings. Particularly the Google Sitemap Generator, since I’ve got to rebuild the sitemap.

13. Checked that site is working

Failed. Because I was using an Ultimate Tag Warrior function for displaying tags. Ok, deleting from my .php files. Site restored, sans tags.

14. Import Ultimate Tag Warrior tags to WordPress

This turned out to be easier than I thought. Look at Technical Itch and Rich’s two part article for more information. Simply go to the WordPress control panel, select [Manage], then [Import], then scroll down to find the Ultimate Tag Warrior link.

Click to enlarge screenshot.

There are 3 steps to the whole thing, although it’s stated there are 5. Maybe it depends on the blog structure. No radio buttons. No check boxes. No typing. Just keep clicking on the [Next] button…

Now to display the tags. Use this format

<?php the_tags('before', 'separator', 'after'); ?>

Cryptic. What before represents is the text before the tags, separator represents the text between tags, and after represents text after the chunk of tags. An example explains this better:

<?php the_tags('Internal tags : ', ' | ', ''); ?>

Which gives something like

Internal tags :  polymath | code | upgrade

Theoretically (I haven’t tried this), you can set it to be

Internal tags: <?php the_tags('<ul><li>', '</li><li>', '</li></ul>'); ?>

to get

Internal tags: <ul><li>polymath</li><li>code</li><li>upgrade</li></ul>

This gives you some flexibility with CSS for a more customised display.

Anyway, that chunk of php code was added into my index.php and single.php, just after the post content. Uploaded the two files and rechecked blog. Yay! We’ve got tags!

One small note: The imported tags have a dash in between words. So if the original UTW tag is “original tag”, it’s displayed as “original-tag”. The original UTW tags are stored with a dash anyway (you should have noticed it when you save your drafts with tags).

It’s not a big deal. Perhaps when I’m in the mood, I might go into the database and flex my SQL powers to do update statements. Right now, I’m just glad everything went smoothly.

15. Write a post

This one’s not in the official upgrade guide. What good is an upgrade if you can’t use it, right? So I went ahead to write up a new post. You’re reading it!

I came across a problem I’ve never come across before. I can’t save my post. What!? After all that hard work, I can’t save a post? Then I checked my post length, which isn’t much longer than usual. I deleted some text and tried. It went through.

Hmm… I added some text and it failed with a 404. The threshold seems to be about 4000 characters, but I don’t think that’s the problem. I did some searching and found this solution about saving with an “Error 404″. Simply add

SecFilterEngine Off
SecFilterScanPOST Off

to your .htaccess file (located at the root of your blog). Problem solved.

[Update] If you still have a problem, try placing the two lines within an IfModule tag, like so

<IfModule mod_security.c>
SecFilterEngine Off
SecFilterScanPOST Off
</IfModule>

Congratulations!

I have successfully upgraded my blog from WordPress 2.2 to 2.3. The entire process took about 3 hours, with the better part of it used on downloading and uploading stuff. Actual manual interaction was much less. I drank tea and dozed lightly in front of the computer most of the time, keeping tabs on the whole process.

I’m so proud of myself. Since it’s hard to pat myself on my back with my hands, I’m going to go get myself a cup of tea instead…

Tell me about your experience with upgrading in your comments!

24 October, 2007 | Written by Vincent Comments Off

Are you helping yourself out of a job?

So I like helping people. In university, during assignment periods when the professor hands out programming tasks, I’m usually one of the first few (if not the first) to complete the programs. My fellow classmates would then pull me (sometimes literally) to their workstations and ask me to debug their programs. After many of these episodes (and the fading feeling of everyone wanting a piece of me), I learned to restrain myself, giving hints and partial answers instead. It’s their grades, and I’m not taking their exams, because I’ve got my own. I felt it’d benefit them more.

Moving on to working life, I found myself, strangely, in a similar position. Not on quite as big a scale as in university though. In one company, there was this guy who really didn’t quite cut it as a programmer. He’s still hired to help the team, so all of us gave our best shot at helping him and easing him into the team.

I remember staying back (together with another colleague) with him till quite late, helping him work through an issue. Alright, I hated it. It was clearly his problem, and there were moments when I really wanted to take over his computer to try out solutions and strangle him, not necessarily in that order.

In another company, I met another programmer who needed help with her work too. The problems were fairly simple, involving the improper use of for loops, if conditions as well as some design logic mistakes. During one of our conversations, I realised something. She was related to the programmer mentioned from before! Surprise, surprise… I kept my discovery to myself and slowly but surely, removed myself from her social radar…

Meeting these people, and helping them, I asked myself, “Am I helping myself out of a job?” Since I’ve started participating in the Dream In Code forums, I’ve had a feeling of being needed again. Recent work life drained a lot out of me, and it felt good to be able to help people with programming problems. Then someone pointed out if newbies will kill the programming star, because we might inadvertently create incompetent programmers who just seem to be competent.

The ideas presented were relevant, and since I’m in the programming profession, I could understand the feelings of frustration and despair. What used to be their problem, became your problem. And after it’s solved, it became their work, while you got nothing.

This actually reminded me of something my professor said (from a few years back). My professor and I were discussing my thesis (computer viral epidemiology), and he said that there were cases where the mentor took credit for a paper his student wrote. Here comes the dilemma for the student. A student, being the unrecognised nobody that he was, needs the mentor (and the mentor’s reputation) to break into the academic circle. How does one claim credit for a paper, if one does not tell anyone? To claim credit, he’d have to tell the mentor and risk his work being stolen. If he doesn’t tell the mentor, he can’t claim credit. Either way, the student loses. Unless the mentor is honest of course.

This is similar to the prisoner’s dilemma problem. Two criminals working together are caught and both are subjected to interrogation separate from each other. Being the idiot criminals that they are, both try to betray the other (and hopefully gain some benefits and leeway), so the good guys get all the evidence needed to put both of them down. If one keeps quiet (or keep to their end of the story), but the other divulge all, then the betrayer gets to go free (or get minimum punishment), while the other gets jail time big time.

If you’re going to be punished anyway, then there’s no reason to keep quiet, is there? Just defect, since the outcome is bad anyway. Well, there’s actually one more outcome, and that’s if both of them cooperate and keep quiet (or keep to the story). Both gets punished, but not as much as if one got betrayed. And I’m not condoning criminal activity here…

Where am I going with all this? If you help someone, your fellow colleague or student or any potential perceived competitor, are you setting yourself to be out of a job?

I just finished watching an episode of “Numb3rs”, where Don, the FBI agent, was visiting a psychiatrist to deal with some personal issues. The topic of leading his team came up, and towards the end of the show, he realised that he’s afraid of letting his team go, of letting his team stand up for themselves. Of his team not needing him anymore. Then he understood that when his team excels, he excels.

Ultimately, I believe helping those fellow programmers improve their skills is beneficial. It depends on the state of mind and belief of the (better) programmer. Cooperate with them more than competing with them. Help one another to make better software. The world needs better software.

And those unscrupulous programmers who are out to get credit yet do little on their own? Fade them slowly out of your life, and concentrate on the people who do appreciate the help you give. Did you know teaching others programming actually make you a better programmer? Because in order to teach something, you have to be great at it.

Or do you think you’re not up to the challenge?

Post your comments! I’d love to hear your side of the story.