Service Oriented Architecture

I was talking with my friend about the direction of software development, and the topic of service oriented architecture (SOA) came up. Having never heard of this term before, I asked for some clarification. Our discussion led me to believe that it’s programming geared towards providing valued services, towards business bottom lines, towards promoting company products.

Boy was I wrong.

I understood it as all efforts centered around the business core services. From top management, to sales, to marketing, to customer service, to programming. Teams are built based on services, say a manager, a customer service staff and a programmer forming a customer facing unit.

When I did some research on Wikipedia to further my understanding, I found my grasp on the subject fatally wrong. I’ll leave you to read up on Wikipedia’s entry on service oriented architecture. There’s a sinking gut feeling as I read the entry. Let me tell you what it was later.

The first sentence already put me to sleep

Service Oriented Architecture (SOA) is an architectural style that guides all aspects of creating and using business processes, packaged as services, throughout their lifecycle, as well as defining and provisioning the IT infrastructure that allows different applications to exchange data and participate in business processes loosely coupled from the operating systems and programming languages underlying those applications.

The second sentence gave a less soporific challenge

SOA represents a model in which functionality is decomposed into small, distinct units (services), which can be distributed over a network and can be combined together and reused to create business applications.

After finishing the entire entry, I found it analogous to an N-tier application. There are user interface screens and business classes and possibly data access components. A new user interface screen is created by slapping on UI controls and running functions from existing business classes. If a new function or new business class is required, then it’s written.

The service oriented architecture simply creates new applications by stringing together existing applications (or services as they’re termed).

Now I have maintained and even coded entire web applications by myself. I know what it’s like to maintain a whole bunch of business classes and know every function quite intimately. And that’s just ensuring the code works cohesively within the web application. It’s hard, but with discipline, it becomes manageable.

But are we talking about inter-application functionality?

I have to admit that this model works well for certain businesses. Social media sites like Facebook and search engines certainly benefit from the publicly exposed API. They probably can survive only if their services are exposed and easily used.

So here’s my gut feeling: It sounds like a lot of work, and the whole thing seems ready to collapse. It also require a lot of control management, as is seen by something further down the entry

using a special software tool which contains an exhaustive list of all of the services, their characteristics, and a means to record the designer’s choices which the designer can manage and the software system can consume and use at run-time.

The concept sounds wonderful. String together a unique set of services in a specific order, and you get a brand new application. I’ve programmed for 5 years, and let me tell you, I’ve never seen an application that can be dismantled into independent and reusable parts for use in creating another application. Projects and requirements are sometimes extremely specific and often need customisation.

Of course, just because I’ve never seen it working doesn’t mean it doesn’t work (that’s a lot of negatives…). I’m just saying that if a business company adopts the model, then someone at a high level has to refactor everything. And I’m not talking about code. Customer service applications, billing applications, inventory applications.

Everything has to be decomposed into independent parts and rewritten as a service. Then someone has to string them up back together to work exactly as the original applications. It’s a huge task if we’re talking about companies offering a diverse set of services and products.

Independent services require structure and well-defined limits and capabilities. It means having the courage to say no. And you know how sales people find it extremely difficult to say that word. There goes billing applications…

I think I prefer the definition my friend and I were discussing…

Practical Programming (part 2)

I’ve talked about the more traditional way of software development and management in part 1. Now I’m going to introduce you to the other extreme: agile software development.

On first instinct, the word “agile” conveys the feeling of flexibility and speed. My first contact with agile software was with extreme programming. I actually had flashes of pallid faces sticking it through the night cranking out code, like extreme sports or something…

There seem to be quite a few variations of what agile software development is. From what I gather, the key points are:

  • Frequent development iterations
  • Focus on customer needs
  • Simple designs

There are a few other points, such as delivering working software in weeks rather than months. I feel the above 3 points most affect how you can be an effective programmer personally as well as when working with others.

Frequent development iterations

The software development life cycle (SDLC) primarily consist of

  • requirement gathering
  • design and development
  • user acceptance test
  • deployment

Those phases are usually very distinct, meaning they don’t overlap in terms of timelines. This makes for very rigid deadlines. If a major bug is discovered during testing, there might not be enough time to correct it.

It also counts as one development iteration. Fine, it just runs once, so it’s not much of an iteration. But in agile development, there are many iterations. They are just much smaller in scale.

Suppose you need to develop 10 independent web forms. Don’t wait until all 10 are done before testing. Finish one, let the user test it while you continue to code the next web form. Then when the user’s done testing on the first one, your second one might be finished.

Perhaps the first one had errors. Perhaps not. Either way, you can continue to code, correcting the first or starting the third.

Hopefully you see the picture. You get feedback almost immediately, and both development and testing are done in tandem. You get a finished product very quickly, even if it’s just one well coded and well tested web form, it’s still a finished product.

The other advantage is that if pressed for time, you always have some kind of finished product at hand. It’s not the full thing, but you get something very close to it.

Focus on customer needs

In part 1, I mentioned that the point of creating software is to make people’s lives better. In particular, you want to make non-programmer lives better. The software is not very useful if you’re the only one who finds it easy to use.

Focus on how other people interact with your software. Focus on your customers because they’re the ones who are in contact with your software everyday.

I’ve found that once I start using my created software from my users’ point of view, what I originally thought was flawless code seemed riddled with inconsistencies.

I talk with many of my users on a fairly regular basis. Through an unfortunate combination of circumstances, I was laden with much maintenance and legacy code, and mostly user interfaced based. What this meant was that the moment users get an error on the screen, my name popped immediately into their minds. And I’ll receive an email or a phone call.

Through the conversations and emails, I gained an understanding of how they went about doing their daily work. And I applied that knowledge to my future development work, incorporating enhancements and design considerations I never would’ve thought of.

And one of my greatest light bulb moments came when I understood the importance of one word…

Simplicity

Make everything as simple as possible, but not simpler.
Albert Einstein

This is one of the most important points in practical programming. The Agile Manifesto says

Simplicity–the art of maximizing the amount of work not done–is essential.

For example, Microsoft Office has many options for doing the same thing. Say you want to save that Word document you’ve been furiously banging on the keyboard for the last half hour. You can click on that diskette save icon. You can do a Ctrl + S. You can click on the menu “File”, then “Save”.

Having options is good. But there’s also the case of having too many options. And the user gets overwhelmed and fails to interact with your software effectively. Why else do you think Microsoft toned down the menu bar and pushed more frequently used items to the forefront in Office 2007?

I develop smaller scaled software. So I stick to a much simpler rule. Make it one option.

There’s only one save button on the web form. There’s only one way to get to the next screen. There’s only one way to retrieve and display data, using the standardly placed “Retrieve” button, which is also standardly named “Retrieve”.

Once I started doing this, my development work simplified. Testing also simplified, since the testers don’t have to test all sorts of saving functions for instance. The users are happy, because they don’t have to think whether they did something correctly, because there’s only one way of doing it, so they must be doing it correctly! And they can continue doing what they’re supposed to be doing: driving the company business.

Closing words

I have obviously a bias towards agile software development. Yet I don’t adopt all the practices some of the proponents teach. My reason? Practicality.

I work in a small team, sometimes being the only developer. As a result, I find it hard to follow any software management method, even one as flexible as agile development.

More than anything, I need fluidity.

In the next and final part, I’ll talk about how I managed a hybrid of the rigid SDLC and the flexible agile models. If you’re a lone wolf programmer, you will find it familiar.

Practical Programming (part 1)

There are many factors affecting the success of a software project. Amongst them, we have

  • Skill level of individuals (from manager to coder)
  • Time involved
  • Tools involved
  • Project complexity

and probably many more you can add. So how can you successfully complete a project practically? How can you balance customer satisfaction, code quality, time constraints and other factors in the most efficient and practical manner?

I was intrigued to explore this when one person asked how can one prepare for a new project and another person asked about end user testing. For those following this blog, you might know that I’m helping out with two teams in my company. Right now, I’m their only .NET (and front-end) developer. My colleagues deal mainly with the backend programs.

I’ve been involved in a start-up company where there was only one other programmer. I’ve also worked with a team of over 10 programmers, of different nationalities, working on a project involving culture localisation in a language none of us were native in (Japanese). Given my current situation, I’m the only programmer, so I’m really interested in practical stuff.

Coming from a mathematical background (as opposed to a computer science one), I don’t know a lot about software development theories and paradigms and stuff. So from what I understand, there is the extreme, full-on software development life cycle (SDLC), complete with UMLs and class diagrams and process flow diagrams and what-nots. There’s usually a lot of documentation involved (I hate those…). There’s usually platoons of programmers involved. There’s usually expensive software and tools and databases involved. The projects are also usually measured in months or even years.

Being mired up to my nostrils in documentation, bureaucracy and meetings with no motivation from completing significant milestones, is not my idea of a fun time.

In this age where the world is getting flatter, companies need to move faster, teams need to move faster. You need to move faster. What happens is that products and services need to be launched faster, earlier and more frequently. And you know what? I don’t think the method of software management mentioned above is going to cut it.

In an age where there’s less time and resources, and you’re expected to create more value, the full life cycle method will drag you down like a 10-ton anchor into the depths of dwindling profitability. Programming practically in this situation requires you to rethink process flows and cut out the fluff. It requires you to be more flexible, more multilingual, more agile.

The point of programming, of creating software, is to make people’s lives better. The resulting software must be practical (or at least fun). The process of creating the software must also be practical (and definitely be fun). So in the next part, we’ll explore the other extreme, of how agile methodologies fit into practical programming.

Piling programmers lead to devastating deadlines

Small red bomb Ever been in a situation where the deadline is getting closer yet the project is getting nowhere? Ever had deadlines looming closer and everyone’s attending meetings, doing discussions but nothing is really done? And the solution from management? Pile on more programmers.

This isn’t like hacking logs, where the more people who chip in to hack logs, the faster all the logs get splintered into firewood.

This isn’t like cleaning house, where the more people who sweep in to mop floors or wipe windows, the faster the house becomes brand new spanking clean.

This isn’t even like data entry, where the more people who come in to type information, the faster all the data gets into the computer database.

Programming is a thinking activity. Thinking activities call on quality, not quantity, of the people involved.

Synchronising actions is easy. Monkey see, monkey do. See log. Grab axe. Swing, splinter, repeat. See dirt. Grab broom. Sweep. Eye on data. Fingers flying on keyboard. Tappity tap, tappity tap. Synchronising thoughts is much harder.

Simply piling on more programmers doesn’t make the project progress any faster. In fact, it just leads to devastating deadlines. Why? It’s the human factor.

This reminds me of a time when I was in a team of 12 (I think). We were working on an enterprise web application (warning bells sounding), since I just joined, I was given documentation of the application framework to study. There were custom web controls, dynamic browser sizing based on user’s resolution, security logins, language resource handling, translations, workflow management class codes and on and on.

I was the new guy, so I was given all sorts of miscellaneous stuff to do while they decide how best to use me. Then I got something to work on. The resulting code wasn’t my best, but I learned a lot from the team and the experience.

Then the deadline started slipping, and management brought on 2 more programmers. While I had some professional experience to speak of, these 2 had hardly any, from what I had seen. She needed help getting a DataGrid sort and bind to work!

New staff, even if they’re experienced programmers, need time to get accustomed to the team and the project.

Me? I got relegated to testing the web application. Somewhere in my resume I stated I did test cases before, and that’s where I ended up, writing test cases. The whole application was finally coming together, and then some of the other colleagues were tasked to help with the testing.

At least management got this part right. Having me help with the coding wouldn’t have made much difference at that point. Testing the application would have smoothed the project’s progress back within deadline, while the last bits of coding were done simultaneously.

Estimating project deadlines is hard enough. Don’t introduce a new variable into the equation. New staff are new variables. And if you really want to hire new variables, make sure they’re polymath programmers.

Estimating software project deadlines

Target dateLet me tell you. It’s hard.

There are managers who never learn why software project deadlines sometimes can’t be estimated with accuracy. There are programmers who never learn how to estimate deadlines with any kind of accuracy. How do we deal with this?

Having an accurate project deadline means schedules can be planned. It means the future can be glimpsed and foretold, albeit somewhat uncertainly. It means we are in control.

Real life doesn’t happen this way. Real world applications are infused with innovation, or they should be for any kind of improvement. This means new concepts, new ways of interacting, new ways of thinking about data, new ways of improving existing processes. New, new, new.

I’ve worked on both long-term huge projects and small-term mini projects. I’ve learned that there are many variables affecting project deadline estimation and the actual project completion date. Some of them are

  • skill of individual programmer
  • understanding of the project/business logic
  • affinity with other programmers on the team
  • difficulty of the task itself

I’m continually given small projects and asked to give an estimate on how long I’d take to complete them. My experience is that the smaller the task, the easier it is to estimate. The more standard the task, the easier it is to estimate.

This doesn’t mean you can’t innovate. On the contrary, it means you must innovate even more. It means breaking down any project into bite-sized portions that you can estimate with certainty. It means you must understand the project on many levels, from how the project fits in with existing projects to how a single line of code fits in.

And it’s hard. Understanding on so many levels require a lot of thinking. It takes some serious brain mojo to figure out that on the macro scale, data is entered and stored in a certain way, and that on the micro scale, a particular if condition has to be coded, even though the condition is never mentioned, implicitly or explicitly.

It involves asking a lot of what if’s. What if the user keyed in this value? What if the user keyed in this combination of values?

The naive manager never understands how a simple project with simple interfaces require so much time, because hidden within the project are assumptions that manager thinks are obvious. The unskilled programmer never learned to gauge his work, because there are assumptions hidden within the project and need to be explicitly coded.

When I’m asked for an estimate, what they’re really looking for is, “How many man days are required?” So far, my answers range anywhere from 5 to 35 man days. In case you’re wondering, 5 man days means 5 work days. It can be 5 people cooperating to finish the task in 1 day, or 1 person doing the task in 5 days. This method gives a projection of the deadline and the cost involved (in paying the programmers).

So how can you improve the accuracy of your estimates? Two factors, skill and standards. Your thinking and programming skills make it easy to translate business logic into code. Which gives you greater confidence in estimating. The more standard a task is visualised, the more previous knowledge can be leveraged. Which reduces new concepts and code, and give you greater confidence in estimating.

Ultimately, it is simply a matter of reducing errors and uncertainty in project estimation. I wonder if L-1 linear regression or least square approximation can be used for projecting future deadlines? I’d have to go through all my documentation to find out what my estimate was and the actual time taken was for any given project… It will require enough data points to sufficiently draw a line.

On that note, FogBugz uses Evidence-Based Scheduling, involving up to 200 data points. Instead of giving you when a project can complete, it gives you a probability of likelihood that the project can complete on a certain date. For example, there’s a 50% likelihood that your project can complete by next week. But if you can extend it another week, there’s a 80% likelihood that the project can complete.

I neither have the time and effort to do least square approximations, nor the software to do historic statistical analysis of previous estimations for future estimations. I do have my experience and programming skill, so basically I trust my gut.

So, how do you estimate your software project deadlines?

5 simple tips to meet project deadlines

Jared was halfway through developing that unbelievably cool application. The specifications were complete, the customer was pleased with the prototype of his user interface and everyone was doing what they did best. There was just this teensy little problem. Sometime ago, Jared had to go help out that fellow colleague with a programming task. Then there’s the time when his users seemed to conspire against him, and decided to take turns calling him with technical queries throughout the day.

Whatever the reason, Jared made a brief calculation and discovered that he’s starting to run out of time. Then he remembered some tips he read from an incredible web site.

1. Shut down your email
Outlook can be detrimental to your productivity. Having to constantly glance at the bottom right of your monitor to check if a new email came in will sap the focus out of you. Ignore people who call you up and say they’ve sent an email 5 minutes ago, and demands that you explain why you haven’t responded. Try checking once in the morning, once after lunch and one last time an hour before you leave work.

2. Go to the toilet often
It matters little if you have a strong bladder. Just go. Walk slower than usual on the way. Have a good look around the office. Take your time while washing up. Your goal is to get away from the computer, away from the task. Your body gets a little exercise and your mind experiences different sensations. All good impetus for igniting your creativity.

If you really want a good reason to go, just follow the next tip.

3. Drink more water
Your brain is already working at hyperspeed. Your body is aching from holding your special fast and furious typing stance. Drink more water. Eat proper foods. Cooling your brain and taking care of your body is important, because you can think better when you can physically keep up with your mental processes.

4. Wear a scowl
Monarch butterflies advertise their distastefulness with contrasting bright colours. Soldiers deter intruders with barbed wire fences. You can achieve the same effect with people who absolutely must bother you with some insignificant detail. Put on the meanest, angriest and most unforgiving scowl you can muster, and note the dropping number of unwanted guests.

5. Leave on time
This may seem counter-intuitive, but leaving work on time forces you to make the best use of your time while you are in the office. You are a responsible programmer, and so you will make sure that the project is still delivered to the best of your abilities. What you want is maximise your throughput instead of increasing the amount of time spent.

Conclusion
After Jared followed those simple tips, he was able to crank out tons of flawlessly elegant code. And lunch with his colleagues. And leave work on time. And manage to catch up with the latest tech news and blogs of his favourite authors.

As Jared was leaving work on the eve of the project delivery, he saw the unfortunate programmers a few cubicles away, mindlessly and desultorily going about their work, having forgone their lunch and now working overtime. Jared laughed (silently of course, as a form of respect for the others), shook his head and left with jaunty steps.

Applying squeeze theorem to software project management

It was a sleepy morning. I was tired and hungry, and in the lecture room, listening to the professor on freshman calculus. He was going on and on, and I was struggling to pay attention, and he mentions the squeeze theorem. Also known as the sandwich theorem. That perked my ears. I was just a tad disappointed when there’s no food involved…

One of the uses of the theorem is to prove that something is equal to something else, say x is equal to 5. If you happen to know that A is less than or equal to x, and that x is less than or equal to B, then you have this

What if A is equal to 5, and B is also equal to 5? You get this

When A squeezes upwards, and B squeezes downwards, by the squeeze theorem, x must necessarily be equal to 5!

Ok, the explanation is a little simple compared to the actual theorem. What I want to point out is that this happens when I’m doing a development project. You see, this is what happens

The customer rarely need everything that they state as a requirement. And the project manager rarely know enough about the software development involved to fulfill what is really required. Three scenarios arise out of this.

They never meet

What happens is I will have to do some serious creative thinking to fill up the gap in the understanding between the two. The customer might want a feature that is hard to implement, but an easier alternative can be done to solve their real concern. The user requirement given to me might have ignored certain software limitations or restrictions, and more effort is actually required. So not only do I have to program a listed feature, I have to make it satisfy both parties when they have different views on the feature.

They overlap

This one is tough to handle, because both parties have their own views on how to do it. My challenge is to prise them apart and convince them that my way will be the best solution. And I have to coerce them in a way that it looks like they are the ones who came up with the solution.

They shake hands

This is the situation I strive for. I have enough specifics to plan the coding, yet have some room for creativity, so as to pleasantly surprise both parties. I have the best chance of success when I am actively involved in the project discussion, or that I can provide preliminary analytical input to one or both parties. Thus will I be able to influence the final requirement.

Sometimes, I have little control over the direction of the software project. Sometimes, I can influence the people involved so that goals are met and everybody is happy with the output. Regardless of the result, I’m going to be squeezed.