Estimating software project deadlines

Target dateLet me tell you. It’s hard.

There are managers who never learn why software project deadlines sometimes can’t be estimated with accuracy. There are programmers who never learn how to estimate deadlines with any kind of accuracy. How do we deal with this?

Having an accurate project deadline means schedules can be planned. It means the future can be glimpsed and foretold, albeit somewhat uncertainly. It means we are in control.

Real life doesn’t happen this way. Real world applications are infused with innovation, or they should be for any kind of improvement. This means new concepts, new ways of interacting, new ways of thinking about data, new ways of improving existing processes. New, new, new.

I’ve worked on both long-term huge projects and small-term mini projects. I’ve learned that there are many variables affecting project deadline estimation and the actual project completion date. Some of them are

  • skill of individual programmer
  • understanding of the project/business logic
  • affinity with other programmers on the team
  • difficulty of the task itself

I’m continually given small projects and asked to give an estimate on how long I’d take to complete them. My experience is that the smaller the task, the easier it is to estimate. The more standard the task, the easier it is to estimate.

This doesn’t mean you can’t innovate. On the contrary, it means you must innovate even more. It means breaking down any project into bite-sized portions that you can estimate with certainty. It means you must understand the project on many levels, from how the project fits in with existing projects to how a single line of code fits in.

And it’s hard. Understanding on so many levels require a lot of thinking. It takes some serious brain mojo to figure out that on the macro scale, data is entered and stored in a certain way, and that on the micro scale, a particular if condition has to be coded, even though the condition is never mentioned, implicitly or explicitly.

It involves asking a lot of what if’s. What if the user keyed in this value? What if the user keyed in this combination of values?

The naive manager never understands how a simple project with simple interfaces require so much time, because hidden within the project are assumptions that manager thinks are obvious. The unskilled programmer never learned to gauge his work, because there are assumptions hidden within the project and need to be explicitly coded.

When I’m asked for an estimate, what they’re really looking for is, “How many man days are required?” So far, my answers range anywhere from 5 to 35 man days. In case you’re wondering, 5 man days means 5 work days. It can be 5 people cooperating to finish the task in 1 day, or 1 person doing the task in 5 days. This method gives a projection of the deadline and the cost involved (in paying the programmers).

So how can you improve the accuracy of your estimates? Two factors, skill and standards. Your thinking and programming skills make it easy to translate business logic into code. Which gives you greater confidence in estimating. The more standard a task is visualised, the more previous knowledge can be leveraged. Which reduces new concepts and code, and give you greater confidence in estimating.

Ultimately, it is simply a matter of reducing errors and uncertainty in project estimation. I wonder if L-1 linear regression or least square approximation can be used for projecting future deadlines? I’d have to go through all my documentation to find out what my estimate was and the actual time taken was for any given project… It will require enough data points to sufficiently draw a line.

On that note, FogBugz uses Evidence-Based Scheduling, involving up to 200 data points. Instead of giving you when a project can complete, it gives you a probability of likelihood that the project can complete on a certain date. For example, there’s a 50% likelihood that your project can complete by next week. But if you can extend it another week, there’s a 80% likelihood that the project can complete.

I neither have the time and effort to do least square approximations, nor the software to do historic statistical analysis of previous estimations for future estimations. I do have my experience and programming skill, so basically I trust my gut.

So, how do you estimate your software project deadlines?

Upgrading WordPress 2.2 to 2.3

UPDATE: The latest versions of WordPress (as of 12 July 2009) all have upgrading options in the administration panel. So you only need to backup your database, click that upgrade link/button in the admin panel, and that’s it. Just go get the latest version, and save yourself the hassle.

So WordPress version 2.3 was released on 24th September 2007. The question is, do you upgrade?

I’m just getting comfortable with WordPress, and then they announce a major upgrade release. As I see it, there are 4 types of people: bleeding-edgers, early adopters, late adopters and the non-budgers. I usually hover between the first two groups. In the case of WordPress, my procrastination could be summed up in one word: laziness.

I’ve been thinking of upgrading for a while already, so finally, one sunny Sunday morning, I went through the whole upgrading process. Here’s what I did (following instructions from the WordPress upgrade page):

1. Backup my database

This one’s easy. The backup instructions provided by WordPress are easy to follow, and there are pictures. Can’t see how anyone can fumble on this.

2. Backup my files

WordPress suggests that I backup all the WordPress files. I thought that since I’m backing up stuff, I might as well go the whole nine yards and backup everything. So I downloaded my entire site onto my computer.

Luckily I’ve got SmartFTP to help me out. I set the download in progress and went for tea. Seriously. The download is gonna take a while…

I also did an export of the posts from within the WordPress panel into the .xml file. Just in case.

3. Verify the backups

Ok, verified the SQL statements are correctly formed in the backup file. I’ve got the entire site on my computer. Moving on…

4. Downloaded all the upgraded plugins

Starting to veer away from the instructions given by WordPress. My reasoning is that I want the upgrade process to happen so fast that even Superman won’t notice any site downtime. So I want everything that can be done without involving the site to be done. This include any downloading of plugin updates.

I’ve got these plugins:

I was able to find updates for all the plugins except Ultimate Tag Warrior, SEO Title Tag and Subscribe to Comments. Well, version 2.3 has in-built tagging capabilities, and I found UTW to be a bit intimidating when I set it up. I don’t need that much flexibility anyway, so UTW is dropped out. It was one of the major reasons why I wanted to upgrade in the first place, for the native tagging capability.

As of this writing, Netconcepts announced a working version of SEO Title Tag for WordPress 2.3 by 31 October. Well, I’m already in the mood for upgrading, so I’ll upgrade later. From what I found, there’s only minimal incompatibilities. Official word from WordPress compatibility results shows that “Primary function is fine but returns a fatal error when tag pages are accessed.” Hmm… oh whatever, so I just went ahead and installed what Netconcepts had first.

As for Subscribe to Comments… no news from Mark, and I made a decision and dropped the plugin. It hasn’t been of frequent use, so it’s not too bad.

5. Downloaded and extracted the WordPress 2.3 package

Same reasoning as above. I mean, what if my Internet connection went limp? You mean I deactivated my plugins, and my site’s sitting there incomplete, and I’m here waiting for my download to complete? No, no, no…

So I downloaded the package first, and extracted everything.

Got a bonus. The stable release is version 2.3.1, so I got something extra. Hope 2.3.2 is a bit further in the future…

6. Deactivate plugins

Now I deactivate the plugins. I was going down the line and when I reached the bottom, I noticed there’s a “Deactivate All Plugins” link provided. Stupid me. Oh well, I didn’t have a lot of plugins anyway…

7. Ensure necessary steps are done

I didn’t follow the WordPress instructions as given. But the idea was to make sure database backup was done (and usable), file backup was done (and usable) and plugins deactivated. Checked.

8. Delete old WordPress files

This one I followed to the letter. I don’t mess around with stuff I don’t fully know or understand. I did make one mistake. I deleted the wp-config.php file. Aahhh! I didn’t realise it until I couldn’t do the upgrade process. More on this later…

9. Upload the new WordPress files

SmartFTP saved the day again. A few clicks and everything’s on their merry way. I went to get another cup of tea…

10. Run the WordPress upgrade program

Basically, you fire up your favourite browser, and type http://example.com/wp-admin/upgrade.php into the address bar (assuming example.com is your site). Or http://example.com/blog/wp-admin/upgrade.php if you installed on a subdirectory “blog” when you first installed WordPress.

This was the point where I realised I couldn’t upgrade because I was missing the wp-config.php file, which I was supposed to leave alone. Luckily, I made a full backup. See how useful that was? Whew… I uploaded the wp-config.php from my backup, and ran the upgrade program again. It worked like a charm!

11. Update Permalinks and .htaccess

Nothing to do really. But I clicked the update button anyway.

12. Installed updated plugins

I uploaded all the updated plugin .php files, and reactivated them. Except for Ultimate Tag Warrior. I went through all the plugins and saved the settings. Particularly the Google Sitemap Generator, since I’ve got to rebuild the sitemap.

13. Checked that site is working

Failed. Because I was using an Ultimate Tag Warrior function for displaying tags. Ok, deleting from my .php files. Site restored, sans tags.

14. Import Ultimate Tag Warrior tags to WordPress

This turned out to be easier than I thought. Look at Technical Itch and Rich’s two part article for more information. Simply go to the WordPress control panel, select [Manage], then [Import], then scroll down to find the Ultimate Tag Warrior link.

Click to enlarge screenshot.
Managing Ultimate Tag Warrior import in WordPress
There are 3 steps to the whole thing, although it’s stated there are 5. Maybe it depends on the blog structure. No radio buttons. No check boxes. No typing. Just keep clicking on the [Next] button…

Now to display the tags. Use this format

<?php the_tags('before', 'separator', 'after'); ?>

Cryptic. What before represents is the text before the tags, separator represents the text between tags, and after represents text after the chunk of tags. An example explains this better:

<?php the_tags('Internal tags : ', ' | ', ''); ?>

Which gives something like

Internal tags :  polymath | code | upgrade

Theoretically (I haven’t tried this), you can set it to be

Internal tags: <?php the_tags('<ul><li>', '</li><li>', '</li></ul>'); ?>

to get

Internal tags: <ul><li>polymath</li><li>code</li><li>upgrade</li></ul>

This gives you some flexibility with CSS for a more customised display.

Anyway, that chunk of php code was added into my index.php and single.php, just after the post content. Uploaded the two files and rechecked blog. Yay! We’ve got tags!

One small note: The imported tags have a dash in between words. So if the original UTW tag is “original tag”, it’s displayed as “original-tag”. The original UTW tags are stored with a dash anyway (you should have noticed it when you save your drafts with tags).

It’s not a big deal. Perhaps when I’m in the mood, I might go into the database and flex my SQL powers to do update statements. Right now, I’m just glad everything went smoothly.

15. Write a post

This one’s not in the official upgrade guide. What good is an upgrade if you can’t use it, right? So I went ahead to write up a new post. You’re reading it!

I came across a problem I’ve never come across before. I can’t save my post. What!? After all that hard work, I can’t save a post? Then I checked my post length, which isn’t much longer than usual. I deleted some text and tried. It went through.

Hmm… I added some text and it failed with a 404. The threshold seems to be about 4000 characters, but I don’t think that’s the problem. I did some searching and found this solution about saving with an “Error 404”. Simply add

SecFilterEngine Off
SecFilterScanPOST Off

to your .htaccess file (located at the root of your blog). Problem solved.

[Update] If you still have a problem, try placing the two lines within an IfModule tag, like so

<IfModule mod_security.c>
SecFilterEngine Off
SecFilterScanPOST Off
</IfModule>

Congratulations!

I have successfully upgraded my blog from WordPress 2.2 to 2.3. The entire process took about 3 hours, with the better part of it used on downloading and uploading stuff. Actual manual interaction was much less. I drank tea and dozed lightly in front of the computer most of the time, keeping tabs on the whole process.

I’m so proud of myself. Since it’s hard to pat myself on my back with my hands, I’m going to go get myself a cup of tea instead…

Tell me about your experience with upgrading in your comments!

Are you helping yourself out of a job?

So I like helping people. In university, during assignment periods when the professor hands out programming tasks, I’m usually one of the first few (if not the first) to complete the programs. My fellow classmates would then pull me (sometimes literally) to their workstations and ask me to debug their programs. After many of these episodes (and the fading feeling of everyone wanting a piece of me), I learned to restrain myself, giving hints and partial answers instead. It’s their grades, and I’m not taking their exams, because I’ve got my own. I felt it’d benefit them more.

Moving on to working life, I found myself, strangely, in a similar position. Not on quite as big a scale as in university though. In one company, there was this guy who really didn’t quite cut it as a programmer. He’s still hired to help the team, so all of us gave our best shot at helping him and easing him into the team.

I remember staying back (together with another colleague) with him till quite late, helping him work through an issue. Alright, I hated it. It was clearly his problem, and there were moments when I really wanted to take over his computer to try out solutions and strangle him, not necessarily in that order.

In another company, I met another programmer who needed help with her work too. The problems were fairly simple, involving the improper use of for loops, if conditions as well as some design logic mistakes. During one of our conversations, I realised something. She was related to the programmer mentioned from before! Surprise, surprise… I kept my discovery to myself and slowly but surely, removed myself from her social radar…

Meeting these people, and helping them, I asked myself, “Am I helping myself out of a job?” Since I’ve started participating in the Dream In Code forums, I’ve had a feeling of being needed again. Recent work life drained a lot out of me, and it felt good to be able to help people with programming problems. Then someone pointed out if newbies will kill the programming star, because we might inadvertently create incompetent programmers who just seem to be competent.

The ideas presented were relevant, and since I’m in the programming profession, I could understand the feelings of frustration and despair. What used to be their problem, became your problem. And after it’s solved, it became their work, while you got nothing.

This actually reminded me of something my professor said (from a few years back). My professor and I were discussing my thesis (computer viral epidemiology), and he said that there were cases where the mentor took credit for a paper his student wrote. Here comes the dilemma for the student. A student, being the unrecognised nobody that he was, needs the mentor (and the mentor’s reputation) to break into the academic circle. How does one claim credit for a paper, if one does not tell anyone? To claim credit, he’d have to tell the mentor and risk his work being stolen. If he doesn’t tell the mentor, he can’t claim credit. Either way, the student loses. Unless the mentor is honest of course.

This is similar to the prisoner’s dilemma problem. Two criminals working together are caught and both are subjected to interrogation separate from each other. Being the idiot criminals that they are, both try to betray the other (and hopefully gain some benefits and leeway), so the good guys get all the evidence needed to put both of them down. If one keeps quiet (or keep to their end of the story), but the other divulge all, then the betrayer gets to go free (or get minimum punishment), while the other gets jail time big time.

If you’re going to be punished anyway, then there’s no reason to keep quiet, is there? Just defect, since the outcome is bad anyway. Well, there’s actually one more outcome, and that’s if both of them cooperate and keep quiet (or keep to the story). Both gets punished, but not as much as if one got betrayed. And I’m not condoning criminal activity here…

Where am I going with all this? If you help someone, your fellow colleague or student or any potential perceived competitor, are you setting yourself to be out of a job?

I just finished watching an episode of “Numb3rs”, where Don, the FBI agent, was visiting a psychiatrist to deal with some personal issues. The topic of leading his team came up, and towards the end of the show, he realised that he’s afraid of letting his team go, of letting his team stand up for themselves. Of his team not needing him anymore. Then he understood that when his team excels, he excels.

Ultimately, I believe helping those fellow programmers improve their skills is beneficial. It depends on the state of mind and belief of the (better) programmer. Cooperate with them more than competing with them. Help one another to make better software. The world needs better software.

And those unscrupulous programmers who are out to get credit yet do little on their own? Fade them slowly out of your life, and concentrate on the people who do appreciate the help you give. Did you know teaching others programming actually make you a better programmer? Because in order to teach something, you have to be great at it.

Or do you think you’re not up to the challenge?

Post your comments! I’d love to hear your side of the story.

Reverse Polish Notation with C#

What is reverse polish notation? It means rewriting a mathematical expression into a postfix notation form. For example, “1 + 5 * 2” is rewritten into “1 5 2 * +”.

Why would anyone want to do that? I actually wrote a C# tutorial on this at Dream In Code, with my explanations there. I wanted to make it really comprehensive, so I edited and reedited my draft. The draft turned out to be more than 4000 words, and that’s not even including the code segments.

Well, apparently the forum engine couldn’t handle my long article, so it’s been shortened. And I really worked hard at the article too…

Anyway, someone at the forum asked something about the Mandelbrot fractal and how to generate colours to make it “pretty”. I gave my take on the matter, but somehow I felt more could be said. It reminded me of a time back in the past …

I was young and full of drive and found the music visualisation in Windows Media Player mesmerising. So I researched on how I could make my own music visualisation software. I needed it to take on different patterns, and I didn’t want to hardcode stuff in.

It was at this point where I stumbled upon reverse polish notation (RPN). RPN allows me to form mathematical expressions and evaluate them in the program code. I was working with C then, so I had for loops and lots of forward and backward checks going through every single character when reading in the expression.

It was terrible. When I got an “s”, I had to check if the next two characters were “i” and “n” respectively, then I could be sure it’s a sine function. I couldn’t find my C code anymore, but I remember that I got it working.

Basically, tokenising the expression was the hard part. Once the tokens were obtained, parsing the expression and evaluating it was easy.

So, back to my tutorial. That question about Mandelbrot fractals got me thinking, “Maybe I can write something on RPN?” With C#, I could use regular expressions to recognise all the different tokens available. So I did that. The thing was, explaining RPN wasn’t the hard part. Explaining the queue and stack data structures, and how regular expressions worked, is. And I used the better part of my tutorial explaining that. We’re going for comprehensive here.

There are two things I left out in the tutorial. One is the use of variables in the expression. The other is adding more functions. Right now only sine, cosine and tangent are supported. The regular expressions used won’t be able to handle hyperbolic sine “sinh” for example. It would fail to recognise whether a token was “sin” or “sinh”. Oh well, maybe another tutorial…

The complete code can be downloaded at the tutorial page.

Blog Action Day 2007 – Use less paper

It’s Blog Action Day, and the topic for this year is the environment. It’s a movement where bloggers around the world blogs about one topic, on one day.

So I was thinking to myself, “What can I talk about, relating to the environment, coming from a programmer’s point?”

I struggled a bit. I’ve thought of recycling bottles and cans. Never really put much effort into it, although I buy bottled or canned drinks infrequently. Using paper bags more often then plastic ones? Yeah, and not nearly enough to quite justify it. Paper? Hmm… now that’s something. I’m accustomed to scribbling down notes on something like an envelope or printing paper or notepad. Then I continue scribbling new notes on those somethings until I can’t find another inch of whitespace.

When I first started working, I was actually appalled at the amount of paper used every month. I was learning the ropes at maintaining an application, and every month, I was to print out several reports and tally the results with each other. After checking that they were correct, I would then file them, and they’d be stuck in some office room corner where they’d never see the light again. Unless a disaster came up, such as a fire or earthquake or some user complaining about incorrect figures.

Then there’s the code. For some reason, the programmers of yore coded with such brutality and inefficiency and unreadability, that sometimes I had to print the code out. I couldn’t make head or tail of what they were trying to accomplish. So I thought holding the code in my hands would give me new insights. It was unbelievably stupid of me. It is in the nature of program code to have lots of whitespace. Sometimes, a curly bracket is all that lies on a single line.

So I tried to be clever, and printed the code as two pages on one side of the paper (you know, the landscape orientation). It was another unfathomably stupid mistake. I reduced the font to size 8 points on Courier New, and the resulting printout was barely visible. I had to squint to make out printf() statements. Absolutely terrible.

I also had the opportunity to attend meetings. Arriving at the meeting room, I’d only have my notebook (the paper kind). Other people would be carrying files of papers and the entire email printout of the meeting notes. And let me tell you, some of the email discussions were very lengthy, and with Outlook doing automatic indents for new replies, text were word-wrapped like crazy. A meeting with ten people probably used enough paper to kill a young sapling.

Another observation I made was the reliance of “hardcopies” of some users. They feel unsafe if there were no printouts of whatever it was they wanted, be it documentation or reports. Reports were printed out even if no one would ever read them. It was the monthly routine, and someone continued to print them out and file them neatly in a nice ring folder. It’s a waste of program resources, human effort and paper.

I understand it if you need to print something out. I’ve printed documents before, like program specifications. It’s just easier to have the business logic printed on paper in your hands, then code on the computer. Alt-Tabbing between a Word document and the Visual Studio is tough, even if I am getting better at it.

What can you do? Simply print less. You don’t have to forgo printing all together. A page saved here, a page saved there, and a tree could have lived a little longer. I’m sure there are parts of the document that you’ll never ever need. So print out the parts that you do need. Sometimes, program specs change, and then what are you going to do? Print it all out again?

Recently, I had the occasion to organise my desk into some semblance of neatness, and I finally decided to do something about a pile of files with documents that were like 10 years old (not mine though). I very nearly took them to the shredding machine and then I stopped. I flipped the document pages over, and they were blank on the other side. I suddenly had an enormous pile of rough paper for scribbling notes! Now to get all the staples out of the way…

I think that ultimately, going all out to save the environment may not be the answer, nor even necessary. Just use less. Use less bottles and cans and paper. Starting small means it’s easier to actually start.

Path of a Polymath Programmer Part 7

I’ve covered diverse topics on my path to becoming a polymath programmer.

  • I went sleuthing with Encyclopedia Brown.
  • I’ve been a warrior, a mystic, a knight, a mutant, a thief, a superhero and many more in role playing games.
  • I went on adventures with computer super spy Orion.
  • I was (and still am hohoho…) phenomenal at typing.
  • I taught myself the Japanese katakana and translated them into English equivalents (whenever possible).
  • I’m well versed in 3D graphics theory and appreciates the beauty in art.
  • I studied computer virus behaviour.
  • I did game development and researched on related topics.
  • I came into contact with the demoscene and appreciates the art and programming involved.

Notice that most of the above had very little to do with programming itself, yet contributed significantly to my development as a programmer. Read up the previous articles at

Your path is most probably very different from mine. The important thing is that you experience, understand and appreciate the many different fields involved. Programming is just a skill. Your ability to understand concepts and translate that into code is the most important thing you’ll ever have.

Joining the workforce

So my malleability and ability to learn fast was tested when I left school. I had to learn how to write resumes and go through interviews. For seven months (yes, I kept track) I was unemployed.

My big break finally came and my recruitment agent called me up one fine afternoon to tell me I’m granted an interview with a local big company. I had just collected my paycheck from my part-time stint as a data entry personnel. Unemployed again, I was wondering what to do next. That call was a lifesaver.

On the day of the interview, I had to call and say I’ll be late. Bad, bad, bad. It was none of my fault, really. The (Mass Rapid Transit) train I was going to take to reach the company had some trouble, and I was stuck at the train station for a while. Still, I was worried out of my skin.

When I reached the company, I found they rescheduled the person after me to get interviewed first. Determined to make the best of the current situation, I calmed myself in preparation for when it’s my turn. The interview turned out ok, and the interviewer seemed impressed by my qualifications in applied mathematics (cool!). I was eventually hired, and on hindsight, I think the interviewer hired me because he believed I’d be a more well rounded addition to his team. His team members were mainly from the computer science and/or computer engineering background.

[short digression]
So, in the interview, his last question turned out to be an IQ question. You were given a cube, and then had it painted on all 6 sides. Then you cut it with 2 parallel and equally spaced slices from the top. Then another 2 parallel and equally spaced slices perpendicular to the 2 from before. And then you cut it horizontally with 2 parallel and equally spaced slices from the side. Basically you’re left with something that looks like a Rubik’s cube. The question? Give a breakdown of the number of smaller cubes thus cut with the respective number of faces painted on those smaller cubes. I’ll give the answer at the end of this post.
[end of digression]

I finally got a programming-related job! My first two weeks turned out to be nothing like what I imagined. I only knew C and Matlab from university. I picked up a little C++ and Windows programming and OpenGL and DirectX on my own. Only the C language turned out to be useful.

The backend programs were written in C. Fine. They connected to the Sybase database. The what? Thus began my first crash course in the language of databases: Structured Query Language (SQL). I learned how to create tables, run select queries, do inserts and updates and deletes.

Then I had to deal with Delphi components. The Windows interface was coded in Delphi and I didn’t know anything about it. Another crash course from a fellow colleague since I’m to support the program.

The .NET phenomenon was still in effect, so the web application interface was written in VB.NET with Visual Studio 2002 (subsequently upgraded to 2003). Didn’t know VB nor VB.NET. Didn’t know Javascript. Didn’t know VBScript. Didn’t know Crystal Reports. It was terrible… and I haven’t even touched on all the business logic and systems and documents I had to go through.

The job changes

Well, after 2 years as a contract staff at the company, I decided it was time to move on. Due to my expertise in the team’s system and business logic, I was too valuable to use for programming, yet not valuable enough to be included in higher profile meetings. I was basically juggling Excel spreadsheets for the users and acting as the helpdesk.

I love programming, and I didn’t like being stuck with mainly administrative and system support work. So I left. I joined a startup company as a C# programmer. I learned first-hand the day-to-day tasks involved with getting the first software product out.

It was also unimaginably unbearable. The stress involved with producing a profitable software application, on the first try, at a startup, overwhelmed me. I prefer a more conducive environment for thinking. So I left, again.

I got hired next as a C# programmer with a software agency. I was then sent to work at another company with the agency’s team that’s already stationed there. The job was to create an enterprise system software (yes your warning bells should be ringing right about now) for the company’s workflow processes.

Despite some of the flaws I encountered, I still learned tons of stuff. I learned about custom web controls, really separating the business logic (like, onto a different web server from the application server) and handling language resources (for an international audience). I learned to work with people from many countries and in a big software team too.

When I reached the end of my 4 month contract, I was actually offered a renewal. But I had other plans. The pay left a little to be desired, and I really wanted to feed myself better. I asked around and got my job from the first company back! I’d be in a different team but still close to my original team. Cool! I’d be experiencing a different work team dynamic.

So at the new team, I was tasked with creating a .NET web application from scratch. There were tons more project deadlines to meet, though the projects were smaller in size. I even got to do some public speaking, presenting (web) applications and training users. And I’m still here as of this writing.

Getting to the blogging

I’ve amassed a ton of experience, both programming and non-programming. And I wanted to share them. I wanted to express my ideas and let people know about them so they can learn from my experience and mistakes. Then I found this site by Yaro about entrepreneurship. He’s also a blogger.

I read through his blog posts and found them really informative. So when he announced that he’s going to teach people how to blog, I signed up right away for his Blog Mastermind program. I mean, a blog is a fantastic vehicle to share my knowledge and experience. Compared with my previous web site creation attempts, a blog is probably easier to maintain.

Well I’ve wanted to be more actively involved in online communities, particular the programming ones. Since Yaro also talks about forums and communities, I figured it’s about time to start. And found Dream In Code. The site offers programming help, has great people and I love the dynamics happening there.

The last word?

So, there you have it, my path of learning and becoming a polymath programmer up till now. It’s by no means complete, because I will continue to learn new things and as a result, be a better person and ultimately be a better programmer.

Share your thoughts!

*The answer to the IQ question above is 1 with 0 sides painted, 6 with 1 side painted, 12 with 2 sides painted, and 8 with 3 sides painted.

Multilingual programming

How many programming languages can you code in? 3? 7? Are you proficient in all of them? “Well, [insert favourite programming language here] is the greatest, and I don’t need anything else!”

Wrong.

If you’ve done any work in a professional capacity, then you’ve probably encountered programs written in archaic languages of old and cutting edge software and everything in between. And sometimes, you need to make them work with each other.

For instance, data could be sent in a text file. It had to be processed with shell scripts in the Unix environment with programs written in C. It’s then dissected and inserted into the database. Then it had to be available in other systems. Problem was, the data in the database wasn’t sufficient. A choice had to be made: expand the data in the database to include more fields, or simply send the raw data to the other systems.

Either way, the other systems were on Windows servers. If raw data was sent, a program had to be written to read it in. Web applications were to be written for data display and manipulation using ASP.NET and C# and VB.NET. So the read-in program might as well be written in a .NET language.

Then comes reading and writing HTML, because even if the IDE was doing a good job of hiding it, sooner or later, looking at HTML would be inevitable. Then there’s Javascript and CSS. Then there’s interoperability with Java programs. And so on and so forth.

Working with different programming and scripting languages, learning the different semantics of operating systems, and reconciling the syntax of different databases. This is what’s going to make you a better programmer, a polymath programmer.

Dream In Code programming community

I can’t remember when it happened, but I had a sudden urge to join an online community. No, not one of those Web 2.0 communities where people Twitter to each other or Digg something. My friends had, over the years, invited me to join some online group or other. Like ICQ, MSN Messenger, then Friendster, MySpace and FaceBook. Never acted on it.

Those groups are like the really social groups. Sharing photos with lots of people. Talking, to lots of people. But I want to talk about programming. I want to talk about math and science and stuff. Somehow, the people around me don’t do this.

So I made up my mind, and went looking for a programming community with compatible interests. After searching high and low (Google helped), I found one that looks promising: Dream In Code. Using some spare time to browse through the site, I got a feel that seemed “right”. It’s a gut feeling thing…

Well, I’ve got to remember a lot of user IDs and passwords, so I’m careful about what I join. It’s a waste of brain cells to think up a user ID and password, only to find you don’t like whatever you signed up for. So after signing up, I went through the forums and tutorials and other pages to really get to know the people there.

The first thing I noticed was, there’s a lot of people asking for help in programming homework assignments. I’m fairly sure these people are students either in high school or university. This is different from the other sites I went to, where most of the problems stem from work related tasks. There are posts about linked lists, prime number calculations and recursion. I just find it unusual that there’s such a high proportion of non-work posts asking for help.

The second thing, and I really like this one, is also unique. It can be summed up as no effort, no code, then no help. The person asking for help must have done something, tried something and when all else failed, post the problem and the code. There must be a specific question to something, like why a certain code chunk doesn’t work, or if the algorithm is correct.

Dream In Code is help-oriented and cooperative. So far, I like interacting with the people there. My handle is “orcasquall”, and I usually browse the C, C++, C#, ASP.NET forums. See you there!

Path of a Polymath Programmer Part 6

I’ve covered a number of topics that have little to do with programming, yet contributed significantly to my programming skills. Here’s a recap of everything so far:

So I’ve been playing console games, interested in pencil-and-paper role playing games and I’m proficient in programming. Is it any surprise that I’d be interested in game development? And that’s what happened.

I browsed through GameDev.net for comprehensive game development guide material. Then I went to NeHe for all the OpenGL tutorials you’ll ever need. I also read up on independent game development on MadMonkey, and browsed GameTunnel for a feel of the games out there.

Since I’m going to be developing games on my own, I’d say I’m an independent game developer, as opposed to the professional game studios such as Square Enix (previously Square). As a hobbyist programmer, I was just looking to create games for fun. At that time, I was interested in creating role playing games similar to Final Fantasy. Researching on game development, I found I had a lot to learn.

There’s the design process, ranging from story plots and characters to weapon and battle system to resource management. There’s graphics to be rendered, and music to be composed. There’s a whole slew of stuff to be done, and all of them had very little to do with programming.

Well, I’ve got to start somewhere, so I began with what I could see, the characters and game world on the screen. I tested the recommended ways on rendering 2D character sprites and how to animate and move them. I wrote small test programs to try out rendering game world data, and camera movement. Luckily I went through a class on 3D graphics, so I could understand terms such as orthogonal/isometric view, point of view (POV) and field of view (FOV).

Then there’s menus. Battle menus. Item selection menus. “New game” or “Load game and continue” menus. I was hovering between using linked lists and simple arrays for storage.

Then there’s resource management. How do I store saved game data? I learned to write data to text files, and read them back and load them onto program variables.

I downloaded and printed tons of research material. I learned the file formats for the Windows bitmap file and music files of the .it (Impulse Tracker), .s3m (ScreamTracker) and .xm (Fast Tracker) file formats.

I learned the basics of design documents and why they’re important. I learned to see the overall view of a game development, which is essential a piece of software. This turned out to be useful in my career.

For a long time, I stuck with it, researching and testing and researching some more. And I still didn’t have anything playable to show for it. I was frustrated and my patience was waning. The only thing holding on was my passion for games, and even that was slowly draining away.

So I did the only thing that made sense. I let go.

Once I freed myself from the obsession that I must create something, I found the whole process more enjoyable. And I didn’t do game development anymore, at least not in the regular sense. I also went back to playing games more, and I started to really appreciate the effort that went into those games.

I still like reading up on game development stuff. I did all that stuff on C, C++, OpenGL and a bit of DirectX (this was where I got that Windows template used in my dissertation in part 5). It was tough because there’s a lot of plumbing that needs to be done. Now there’s pixel shading, anistropic filter and much better graphics cards. Now there’s C# and XNA Game Studio Express from Microsoft. Aha! Useful tools taking out a lot of the mundane and bringing back more of the fun. Maybe I’ll try my hand at creating something again…

The next obsession

Well, game development took too much out of me. But I continued to browse the game development web sites. Then I came upon a site where the author listed some of his favourite applications, games or otherwise. And that’s where I met “The Product“.

“The Product” is a demo. It is an application that runs for about 11 minutes, showcasing 3D scenes, special effects and accompanied with synchronised music. It is also 64 kilobytes in size.

That was astonishing! After my studies of game development, I realised that lots of game data was used. Even console games require memory cards to store saved game data. Then came along one small piece of software that creates an amazing visual and auditory experience. That just blew my mind away. Absolutely cool.

Creating demos is part of a culture called the demoscene. It had a rather unsavoury historic start, because it began as attempts to hack into programs and load some of the hacker’s taunts into that program. It was used to demonstrate the superiority of the hacker’s programming skills. The hack had to be small in size, and tons of creativity went into making the hacks as memorable (and insulting) as possible.

Well, the demoscene has evolved into something else. Something more artistic in nature. To learn more about the scene (as it’s called), I suggest actually downloading the demos and have a look. You can try these 5 amazing demos first. Then go to scene.org and pouet.net for more information.

Anyway, where was I? Oh yes, the product. I wanted to find out how it was done. All the textures. All the music. All the 3D scenes rendered. How were they stored?

It turns out that the demogroup who created it, wrote generic functions for generating textures, music and 3D models, right in the application. Like a class library of sorts. Then in the application, only the function calls are stored. And that’s how all that data was stored in just 64 kilobytes. You don’t store the data. You store the instructions for creating the data. Brilliant.

After watching the product, I wanted to watch more. I downloaded lots of other demos and watched them, over and over again. The special effects were generally better than the games I played. The music were generally of high quality, some mesmerising and some blood-pumpingly intense.

And then I wanted to create a demo of my own. Ha! Man it was tough. I started researching again. I found that there’s an electronic magazine about the demoscene called Hugi. It’s basically an application that runs and lets you browse through articles in an interactive manner, with music playing in the background. These people are so awesome…

With much passion and enthusiasm, I actually finished a demo. It’s called “Trying To Fly”. It’s about this wooden cuckoo bird wanting to escape into the bigger world, trying all sorts of methods to actually fly. Well, it never took off, pun not intended. BUT, I did learn the process of switching scenes, animation of 3D models and particle effects. Yeah, I wrote a custom particle system just for the demo. It’s actually kind of cool, and I think I’ll talk about it in another post.

Continue to part 7