The math behind 360 degree fisheye to landscape conversion

I wrote an article to convert a 360 degree fisheye image to a landscape view some time ago. I also realised I didn’t explain the math very much, mainly because I thought it’s fairly obvious. On hindsight, it doesn’t seem obvious. My apologies.

Commenter Eric pointed out a math technique called linear fractional transformation. It’s basically a function that can map lines and circles to lines or circles.

In theory, it seems applicable to our problem. When I tried working out the solution, I failed. 2 possible reasons: my math sucks, or the technique isn’t applicable to the problem. It’s probably the former…

My postulate is that, the fisheye-landscape conversion has an edge condition that maps a line to a point. Specifically, the bottom of the destination image maps to one point, the centre of the source image. Thus linear fractional transformation is probably not suitable. I’m happy to hear from you if you’ve found a way to make it work.

Let’s bring up the explanation diagram I had from before:

Fisheye to landscape explanation diagram

I have assumed a source image that’s square (width equals height) and its width has an even number of pixels. With this, let the given image be of width and height of 2l pixels. The idea is to construct an image with width of 4l pixels and height of l pixels, which is the landscape view of the source fisheye image.

The dimensions of the destination image was arbitrarily chosen. I just find that a height of l pixels (or half the height of the source image) to be convenient. The centre of the source image is assumed to be the centre of the small “planet”. This means the pixels along the horizontal and vertical bisectors of the source image will not be distorted (much) by the conversion.

Oh right, I haven’t told you about the exacts of the conversion math…

You should know that only the pixels in the inscribed circle of the source image would be in the destination image. This is due to the “uncurling” effect. The pixels not in the inscribed circle would be out of range in the destination image.

So, imagine the source image as a representation of the Cartesian plane. The centre of the image is the origin. The point A, is the eastern point of the inscribed circle. Points B, C and D are the northern, western and southern points of the inscribed circle respectively.

I’m using the Cartesian plane because the Cartesian quadrants make the math easier. Circles mean sines and cosines, so I’d rather work with angles in the typical form than do all the funny angle modifications. I’m not masochistic, you know…

What you should understand now is this: the pixels along the top of the destination image come from the pixels along the circumference of the inscribed circle on the source image.

We’ll be iterating over the destination image (remember my messy index preference?) Let’s start at the top left corner. We’ll be iterating 4l pixels to the top right corner. This is visualised as going clockwise on the source image from point A, to D, to C, to B and back to A.

So, 4l pixels is equivalent to 2 PI radians?

At the top left corner, we start with 2 PI radians (so to speak). As we iterate to the top right corner, the angle reduces to 0. Thus this formula:

theta = (4l – x)/(4l) * 2PI
where x is the Cartesian x axis.

Generally speaking, iterating from the left to right on the destination image is equivalent to going clockwise on the source image.

Now, as we iterate from the top left of the destination image to the bottom left, it’s equivalent to going from point A on the source image to the centre of the source image. Thus:

radius = l – y
where y is the Cartesian y axis.

Generally speaking, iterating from the top to bottom of the destination image is equivalent to going from edge of source image to centre of source image.

And once you understand that, the rest is just coding. I merged the code for converting Cartesian to raster coordinates together (more details here with code here). The code was deliberately left unoptimised so it’s easier to read.

For example,

theta = 2.0 * Math.PI * (double)(4.0 * l - j) / (double)(4.0 * l);

could be

theta = Math.PI * (double)(4.0 * l - j) / (double)(2.0 * l);

to save on operations. But the 2.0 * Math.PI makes it more meaningful.

The if condition

if (x >= 0 && x < (2 * l) && y >= 0 && y < (2 * l))

could have had the (2 * l) part assigned to a variable to avoid multiplication multiple times. You're welcome to use a variable perhaps named iSourceWidth for example.

And that's all I have. I hope you have fun with the code.

The leap year 1900 “bug” in Excel

No it’s not really a bug in Microsoft Excel. What happens is that Excel will accept 29 Feb 1900 as a valid date.

“Wait, the year 1900 is not a leap year. 29 Feb 1900 is invalid!”

Yes, I agree. I wrote something about leap years before. From what I understand, it’s a historical issue, that

There are two kinds of Excel worksheets: those where the epoch for dates is 1/1/1900 (with a leap-year bug deliberately created for 1-2-3 compatibility that is too boring to describe here), and those where the epoch for dates is 1/1/1904.

[emphasis mine]

The “1-2-3” refers to Lotus 1-2-3, a spreadsheet program. To get the Lotus users to come over to Excel, Excel had to be able to import files in the Lotus format. Unfortunately, leap years weren’t well understood then, so Excel used the wrong leap year calculation, as did other spreadsheet software, which Microsoft acknowledges.

I’m talking about this because I’m creating Excel files in the Open XML format (for reporting purposes). In particular, I work with dates. For example, I have to create reports for call detail records that pass through satellites for my users. The date value is one of the most looked at piece of information.

Anyway, to study how dates are stored in Excel, I created an Excel file with date information such as “26/10/2009 12:34” and saved it. I used to save it in the (Excel 2003) XML format, but with Open XML and the Microsoft Office Compatibility Pack (for Office 2003 and earlier versions), I tried the .xlsx format.

I renamed the .xlsx to .zip (because Open XML files are just zip files), then unzipped the package. I looked at /xl/worksheets/sheet1.xml and looked for my date data. It disappeared! They’re just floating point numbers!

Actually, those floating point numbers represent the number of days since the epoch 1 Jan 1900.

How did I discover that? I can’t remember the details. I think I printed consecutive days such as 1 Jan 2009, 2 Jan 2009 and so on, and then checked the floating point number in the resulting XML file. I found that they differed by a difference of 1.

Then I made the brilliant mother of all light bulbs and postulated that perhaps the floating point numbers started counting from an epoch. The only epoch worth my while was 1 Jan 1900.

So I amended my test program to start generating date values
01/01/1900
02/01/1900
03/01/1900
04/01/1900
05/01/1900
and so on.

I looked at the resulting XML file and saw 1, 2, 3, 4, 5 and so on. My next brilliant bit of deduction was that the trailing decimal values must be fractional values of days. Pleased with myself, I proceeded to test that theory.

It worked fine. Till I hit 29 Feb 1900. My instincts warned me, “That doesn’t look right.” Generally, I look out for edge cases, and in the case of dates, 29 Feb. I did some calculations and woah, 29 Feb 1900 isn’t valid! Yet there it was in the Excel file, displayed and formatted correctly by Excel.

That’s when I dug around and found Joel’s article.

So how do you fix it?

DateTime dtEpoch = new DateTime(1900, 1, 1, 0, 0, 0, 0);
DateTime dt = DateTime.ParseExact("05 Mar 1900", "dd MMM yyyy", null);
TimeSpan ts = dt - dtEpoch;
double fExcelDateTime;
// Excel has "bug" of treating 29 Feb 1900 as valid
// 29 Feb 1900 is 59 days after 1 Jan 1900, so just skip to 1 Mar 1900
if (ts.Days >= 59)
{
	fExcelDateTime = ts.TotalDays + 2.0;
}
else
{
	fExcelDateTime = ts.TotalDays + 1.0;
}

If the date in question is on or after 1 Mar 1900, simply add one to the floating point value.

You might wonder why we added one more to the value in each section of the if-else portion. If the date is on or after 1 Mar 1900, shouldn’t it be +1.0 instead of +2.0?

Because it’s a counting problem. We excluded the epoch date itself when we calculated the TimeSpan ts, so we’re adding it back in.

You might wonder why you couldn’t have used 31 Dec 1899 as the base. You can. I just think 31 Dec 1899 doesn’t quite ring a bell. 1 Jan 1900 is more striking.

“But I don’t use the year 1900 at all! I mean, even the year 2000 is, like, so last century. The year 1900 is like, Babylonian!”

Hmm… ok. *shrugs*

There’s also updated material and source code, together with more information on how to work with Open XML. Click here to find out more.

Figuring out who you are

You have to watch this video first. It’ll be one of the most thought-provoking 40 minutes of your life:

Makebelieve Help, Old Butchers, and Figuring Out Who You Are (For Now) from Merlin Mann on Vimeo.

I’m scared. I’m deathly afraid actually. Remember the ebook I’m writing, “Discipline and Deflection”?

Well, I started off thinking, “I want to help people. I seem to have a knack for handling many small tasks, answering emails, replying to user queries, generating ad-hoc reports from databases. Stuff like that. And I still manage to write code, roll the applications out to production, and maintain legacy code. I should totally write something about handling interruptions!”

That went off to an immense spurt of creative energy. I couldn’t stop thinking about it. Maybe I could write that, what reference could I use to illustrate my point and so on. I would be jotting down notes on my paper pad. I would be typing in tidbits of inspiration on my iPhone if I’m travelling.

As I thought and planned and wrote and thought some more, I had this increasing feeling of “This is so lame! How could anyone possibly benefit from this? This doesn’t make sense!”

And I got scared. What the heck was I doing? How could I possibly think I’m good enough to teach anyone about handling interruptions?

So I changed course. I thought about it real hard, and hit upon the idea that I’m not really writing about handling interruptions per se, but about controlling one’s self. It’s about self-control. It’s about controlling the emotions you’re feeling, thoughts you’re having and actions you’re taking. It’s not about suppressing those emotions, nor feel guilty about having flighty fantasies, nor getting depressed over the things you’ve done. It’s about not reacting to your emotions, which govern your thoughts, which compel your actions.

It’s about self-control, about self-awareness, and about how to use that knowledge to better yourself, in math, in programming, in whatever you set your mind to do.

And I got really scared after that. Because I felt like I was becoming one of those fake self-help people Merlin was talking about in his video. Because that piece of work I have in my computer right now, that ebook I’m going to publish, has little to do with math. Or programming. The concepts’ relation to each other seem as weak as molecular bonds.

But I used them. I figured them out myself. And they worked for me. But that doesn’t make me an expert in any way. And so I’m afraid. Of sounding like I know what I’m doing. I don’t.

Mostly, I just intuit things to their conclusion. This seems to run counter to the logical processes that are math and programming. I use intuition and logical thinking in tandem and in equal parts to solve problems.

I was so scared that while writing this article, I had to get away from the keyboard, and since I was also hungry, I mixed up a protein shake to drink (it was about 11pm and I didn’t want to eat anything heavy). The act of mixing, drinking the shake, and washing up calmed me a little. Which is important.

Because…

Most of all, I’m scared because I have no idea how that piece of work is going to help you. I know you’re a smart person. Which adds to my fear. What can I possibly teach an intelligent person on how to stay calm, how to maintain discipline, how to follow through on the task at hand even with interruptions and distractions? Will that even help you in your work?

In any case, I’m still going to finish that ebook. Even if it doesn’t help you. Even if you think it’s lame. Even if you think that makes me a fake self-help guru.

Because I have a compulsion to finish it. I have to write it. It’s driving me insane.

I’m going to stop here. I know you probably have some really useful work you need to get done. Still, a comment or an email with your thoughts on the matter is very much appreciated. Thank you.

English Metric Units and Open XML

Emu
[image by blmurch]

In this article, you’ll find out how English Metric Units (or EMUs) are related to the Open XML format. So what are English Metric Units? They are … actually, I have no idea what they are. Here’s the best answer I could find on EMUs. And there’s the Open XML explanation on EMUs in Wikipedia.

I stumbled upon EMUs because I was trying to create an Excel spreadsheet using the Open XML format. The task require the addition of an image in one of the Excel sheets. You have no idea how much code I needed to write just to include one image. (I’ll tell you about that in another article. Let’s focus on EMUs here.)

Basically, there’s no unifying unit, no “one ring to bind them all”: centimetres, inches and points. So they used a new unit of measurement to represent the dimensions of an image. *sigh*

Anyway, while searching for how to convert pixels to EMUs, I stumbled upon 2 articles on StackOverflow: Pixel to Centimetre and Pixel to Point.

From the Wikipedia article, there are 914400 EMUs per inch, and there are 96 pixels to an inch. Therefore, I figured the pixel to EMU formula is

EMU = pixel * 914400 / 96

There’s a flaw, in that the dots per inch (DPI) may be different for different monitors. For example, there could be 72 pixels in an inch. The best I could do is to assume that, the image created with one monitor, will be used on another monitor with the same DPI. Thus:

EMU = pixel * 914400 / Resolution

Here’s some code to visualise that:

Bitmap bm = new Bitmap("yourimage.jpg");
DocumentFormat.OpenXml.Drawing.Extents extents = new DocumentFormat.OpenXml.Drawing.Extents();
extents.Cx = (long)bm.Width * (long)((float)914400 / bm.HorizontalResolution);
extents.Cy = (long)bm.Height * (long)((float)914400 / bm.VerticalResolution);

There, now you know how English Metric Units are related to Open XML formats.

There’s updated material and source code, together with more information on how to work with Open XML. Click here to find out more.

P.S. English Metric Units have no relation to emus. Other than their (unfortunate?) acronym being exactly the same as the name of the Dromaius novaehollandiae

Should you be a generalist or specialist?

The short answer is: be both.

That seems to cheat you of an interesting discussion, so let’s talk more on it. Let’s define the terms first. Generally speaking (that’s not a personal bias, that’s how the language is used…), a generalist is a person whose interests and skills are varied. A specialist is a person who’s very interested and/or skilled in 1 (maybe 2) area(s).

So why are specialists highly sought after? As a rough benchmark, let’s use Google Trends:
Generalist vs Specialist trend

One possible reason is that people feel safer going to a specialist for help. For example, people would rather see a lung specialist, or heart specialist, or kneecap specialist than a general practitioner. Even for small problems.

I’m not saying it’s wrong. It makes sense.

I’m saying, as the world grows more complex, so does its problems. As the complexity grows, it becomes harder to categorise those problems. Thus making it difficult to know who to consult.

Say the web application has some errors. Is it Javascript? Is it improperly formed HTML? Is it the CSS? Is it the database permission? Is the database down? Is the server down? Is the code wrong? And are you telling me you need to look for a Javascript expert, a web designer, a database administrator and a programmer in order to fix that problem?

The general consensus (there’s the word again) seems to be to focus on your strengths and correct your weaknesses as best as possible (or completely ignore them).

As the world’s problems grow more complex, I propose that the categories start to merge together. You don’t think of them as Javascript or CSS or database problems. You look at them as a web application problem. And you look for someone specialising in developing web applications.

So should you be a generalist or a specialist? I would say be both if you can. The world needs both. As Seth Godin says, specialise in being a generalist.

It feels like an intractable proposition. How do you know many things and know one thing very well at the same time? How do you stand still and move at the same time? How do you clench your fist and unclench your fist at the same time?

Well, don’t think of them as opposite sides of a solution. Think of them as the same solution.

Convert 360 degree fisheye image to landscape mode

In this article, you will learn how to flatten a 360 degree fisheye image back to its landscape panoramic form. But first, what’s a 360 degree fisheye image?

[WARNING: graphic-intensive article ahead]

Lillestrøm in fisheye mode
[image by gcardinal]

It’s created by using fisheye lens in your camera, take a bunch of pictures using that, and use stereographic projection to get the final image. Or so I understand.

Basically, it’s like taking a normal (but panoramic is better) picture and “curling” it along its bottom or top (but much more complicated than that).

Here’s how to visualise the fisheye image. Hold your right hand, palm downwards, in front of you, thumb towards you, little finger away from you. The tips of your fingers form the “left end” of the image. Your wrist forms the “right end” of the image. Now form a fist with your right hand. There’s your fisheye image.

The conversion algorithm explanation

So given the fisheye image, we want to get that landscape image back. And the way we do that, is to “uncurl” the fisheye image. Here’s the diagram to explain the logic:

Fisheye to landscape explanation diagram

You may have noticed that the corners of the source image will not be in the resulting image. You may also notice that the centre of the source image is very “squeezed”, and the pixels around there will be repeated in the resulting image.

The first problem can be solved by using a bigger destination image. But it doesn’t matter to me, and you will get a jagged top with unfilled pixels. I didn’t like that, so decided to give those up.

The second problem… I don’t know if there’s a solution. Because you’re trying to “guess” the pixels mapped in the destination image from the source image. But the source image has less pixel information. The simplest solution seems to be to get a higher resolution source image. But that only mitigate the problem, not solve it.

You may also notice that only the pixels within the inscribed circle of the source image is used. Well, what do you get when you curl up a line? A circle. *wink*

What happens when circles are involved? Radius and angles, that’s what.

So in the destination image, in raster coordinates, going from top to bottom is equivalent to going from outer inscribed circle of source image to centre of source image. Or the variable l slowly reduces to zero.

Going from left to right is equivalent to going from 2PI radians and decreasing to 0 radians on the inscribed circle. It’s also equivalent to going from 0 radians to -2PI radians. Sines and cosines are periodic functions.

Here’s another diagram to show what happens when we iterate over the destination image:

Before we get to the code, here are 2 assumptions to simplify the process:

  • The source image is square
  • The width of the source image is even

They’re not necessary, but they make the programming easier. And I’m mapping the quadrants to the standard Cartesian quadrants because they make the math easier. The centre of the image should be the centre of the “circle” (or that small planet, as it’s affectionately known).

[The original source image wasn’t square, and its centre wasn’t the centre of the planet. So I cut the image, and guessed as best as I could on the centre. More info on the centre later in the article.]

Fisheye to landscape algorithm/code

I’m plagiarising my own code from the image rotation with bilinear interpolation article for the bilinear interpolating parts. There are 2 resulting images, one with and one without bilinear interpolation. And here’s the code:

// assume the source image is square, and its width has even number of pixels
Bitmap bm = new Bitmap("lillestromfisheye.jpg");
int l = bm.Width / 2;
Bitmap bmDestination = new Bitmap(4 * l, l);
Bitmap bmBilinear = new Bitmap(4 * l, l);
int i, j;
int x, y;
double radius, theta;

// for use in neighbouring indices in Cartesian coordinates
int iFloorX, iCeilingX, iFloorY, iCeilingY;
// calculated indices in Cartesian coordinates with trailing decimals
double fTrueX, fTrueY;
// for interpolation
double fDeltaX, fDeltaY;
// pixel colours
Color clrTopLeft, clrTopRight, clrBottomLeft, clrBottomRight;
// interpolated "top" pixels
double fTopRed, fTopGreen, fTopBlue;
// interpolated "bottom" pixels
double fBottomRed, fBottomGreen, fBottomBlue;
// final interpolated colour components
int iRed, iGreen, iBlue;

for (i = 0; i < bmDestination.Height; ++i)
{
	for (j = 0; j < bmDestination.Width; ++j)
	{
		radius = (double)(l - i);
		// theta = 2.0 * Math.PI * (double)(4.0 * l - j) / (double)(4.0 * l);
		theta = 2.0 * Math.PI * (double)(-j) / (double)(4.0 * l);

		fTrueX = radius * Math.Cos(theta);
		fTrueY = radius * Math.Sin(theta);

		// "normal" mode
		x = (int)(Math.Round(fTrueX)) + l;
		y = l - (int)(Math.Round(fTrueY));
		// check bounds
		if (x >= 0 && x < (2 * l) && y >= 0 && y < (2 * l))
		{
			bmDestination.SetPixel(j, i, bm.GetPixel(x, y));
		}

		// bilinear mode
		fTrueX = fTrueX + (double)l;
		fTrueY = (double)l - fTrueY;

		iFloorX = (int)(Math.Floor(fTrueX));
		iFloorY = (int)(Math.Floor(fTrueY));
		iCeilingX = (int)(Math.Ceiling(fTrueX));
		iCeilingY = (int)(Math.Ceiling(fTrueY));

		// check bounds
		if (iFloorX < 0 || iCeilingX < 0 ||
			iFloorX >= (2 * l) || iCeilingX >= (2 * l) ||
			iFloorY < 0 || iCeilingY < 0 ||
			iFloorY >= (2 * l) || iCeilingY >= (2 * l)) continue;

		fDeltaX = fTrueX - (double)iFloorX;
		fDeltaY = fTrueY - (double)iFloorY;

		clrTopLeft = bm.GetPixel(iFloorX, iFloorY);
		clrTopRight = bm.GetPixel(iCeilingX, iFloorY);
		clrBottomLeft = bm.GetPixel(iFloorX, iCeilingY);
		clrBottomRight = bm.GetPixel(iCeilingX, iCeilingY);

		// linearly interpolate horizontally between top neighbours
		fTopRed = (1 - fDeltaX) * clrTopLeft.R + fDeltaX * clrTopRight.R;
		fTopGreen = (1 - fDeltaX) * clrTopLeft.G + fDeltaX * clrTopRight.G;
		fTopBlue = (1 - fDeltaX) * clrTopLeft.B + fDeltaX * clrTopRight.B;

		// linearly interpolate horizontally between bottom neighbours
		fBottomRed = (1 - fDeltaX) * clrBottomLeft.R + fDeltaX * clrBottomRight.R;
		fBottomGreen = (1 - fDeltaX) * clrBottomLeft.G + fDeltaX * clrBottomRight.G;
		fBottomBlue = (1 - fDeltaX) * clrBottomLeft.B + fDeltaX * clrBottomRight.B;

		// linearly interpolate vertically between top and bottom interpolated results
		iRed = (int)(Math.Round((1 - fDeltaY) * fTopRed + fDeltaY * fBottomRed));
		iGreen = (int)(Math.Round((1 - fDeltaY) * fTopGreen + fDeltaY * fBottomGreen));
		iBlue = (int)(Math.Round((1 - fDeltaY) * fTopBlue + fDeltaY * fBottomBlue));

		// make sure colour values are valid
		if (iRed < 0) iRed = 0;
		if (iRed > 255) iRed = 255;
		if (iGreen < 0) iGreen = 0;
		if (iGreen > 255) iGreen = 255;
		if (iBlue < 0) iBlue = 0;
		if (iBlue > 255) iBlue = 255;

		bmBilinear.SetPixel(j, i, Color.FromArgb(iRed, iGreen, iBlue));
	}
}

bmDestination.Save("fisheyelandscape.jpg", System.Drawing.Imaging.ImageFormat.Jpeg);
bmBilinear.Save("fisheyebilinearlandscape.jpg", System.Drawing.Imaging.ImageFormat.Jpeg);

So what did we get from our fisheye image?

Let’s look at our results, shall we? First, let’s bring our source image back.

Lillestrøm in fisheye mode

Here’s the straight-up result from the algorithm:

Lillestrøm in landscape mode
[click for larger image]

Here’s the result with bilinear interpolation:

Lillestrøm in landscape mode with bilinear interpolation
[click for larger image]

And I’m fortunate that the original photographer had the original landscape version for comparison:

Lillestrøm
[image by gcardinal]

Hmm… better than I expected. The code was also easier than I expected. I think it’s the math that was harder…

The fisheye image I used is one of those where the bottom of the original landscape image is curled into the centre. The other type is where the top of the original image is curled into the centre.

In that case, using the algorithm provided results in an upside-down image. I’ll leave it as an exercise to you for flipping the resulting image right-side-up.

One note about the centre of the source image. I found that if the centre is off, the horizon of the resulting image won’t be flat. I’m just showing you how to get the landscape image from a “perfect” source fisheye image. I’ll leave the details of fixing missing corners and undulating horizons to you.

Oh, while figuring out the algorithm and writing the code, I needed a test image. So I made this monstrosity:

Fisheye test image

with this result:

Fisheye test landscape image
[click for larger image]

It ain’t pretty, but I needed to test if the quadrants map correctly…

P.S. A reader submitted this in a recent survey. I hope I’m interpreting the question and answering it correctly.

All systems are in a state of entropy

I’m surprised to find multiple meanings for the word “entropy”. I thought it just meant “breaking down” or something. I was searching online and doing research, and … well, even though I quoted:

Google not synonym for research

Anyway, it occurred to me that all systems are in a state of entropy (we shall use the version of “breaking down” henceforth). A tree is in a state of entropy. A lump of rock is in a state of entropy (the molecules are breaking down, just very very slowly). The rubbish in land fills is breaking down. You are breaking down (the molecular deconstruction part, not the crying part).

Yet all seems fine. The tree is still standing. That lump of rock is still showing the world how durable it is. The rubbish in land fills is still … decomposing (unfortunately not fast enough). And you are still here (thankfully).

The virus behaviour simulation program

The young man was stuck. He needed to write a thesis paper to complete his degree qualifications, but he didn’t know what to write about. The topics available weren’t interesting enough, so he consulted his thesis advisor. And the advisor said, “Why don’t you write something on computer viruses?”.

And that’s what the young man did. And to complement the research, he would write a program to simulate how computer viruses spread and behave, based on his research.

He wrote a preliminary program to do simple “infection” and “disinfection” of nodes (representing computers). The node system became Resident Evil after a few iterations.

Then from his research, he learned about topography. So he applied a few of them. The results were interesting, and one showed promise: the tree structure.

Did more research (and worked on his thesis, solving some integration equations on the way), and thought of the Internet as a fragmented tree structure. And so he created a new topological structure based on uneven connectedness.

That was promising. But the stabilising of computer viral infection was too slow. Something was missing…

And then he came upon the brilliant idea of periodicity. Maybe computers weren’t “on” all the time! He factored that in, and the resulting “epidemic” simulation finally approached what looked like a sample chart from a well known virus protection software provider.

Just keep repairing

I was thinking of the general sentiment “Every program has bugs”. Every program is imperfect, even the Hello World program. Why is that so?

Because even programs obey the entropy rule. The moment you think up the code for the program, it’s already on the way to decay.

In the case of the story above, the simulation program was “buggy”. As new information surfaced from the research, that knowledge was used to improve the simulation logic. But it’s nowhere near done. [In it’s present state, the simulation is probably so far off the mark, that I’m … uh, I mean the young man is ashamed of it.]

You may think it’s done, it’s perfect. There shouldn’t be any bugs, or any changes that will make it better. But it’s not.

Because there are always patches on the most reliable of software. Someone always manages to find a vulnerability to exploit.

Perhaps the program works fine, but its environment is “decaying”. The data it fed on changed slightly, causing its input mechanic to fail. The hardware it’s running on blew a small circuit, causing tiny imperfections on a physical level, which caused it to choke when it ran certain code constructs.

So if all systems are in a state of entropy, even software, what can you do? Keep repairing them. Keep improving them.

If you’re not the negative type, then all systems are in a state of change. But are they changing for the worse, or better?

Here’s a question for you to think about. Is an idea in a state of entropy?

P.S. I just finished reading The Lost Symbol by Dan Brown. If you’re not the philosophical type, I apologise. And then I blame Dan Brown.

Discipline and Deflection

Meditating by the sea

“Discipline and Deflection”. That’s the title of the ebook I’m working on. I started with the idea that perhaps, just maybe, I could write about how I dealt with (coding and non-coding) interruptions while still completing projects (I wrote a bit on that here). That I could help you, in case you happen to want to know more about that.

As I thought and consolidated my points, every single point seem to revolve around the idea of self-control. Without self-control, any life-hacking, GTD-esque, productivity tool you have is useless. Because without self-control, you won’t have the discipline to use those tools and actually do what you wanted in the first place.

When you have something important to do, you need to focus. This actually has 2 parts. You need to concentrate on that one task, and ignore any interruptions (including other tasks). It turns out that there’s lots of advice on this, and that it’s hard.

So I don’t have like “47 tips on making your browser work better”, or “12 essential tools you must have on your computer” or whatever number that’s popular right now. I only have 1 (which is fine. I mean it’s the number one!), or 2 (it’s the only even prime number!) things to tell you.

It’s just self-control. And there are 2 parts: discipline and deflection. Hard and soft. Yang and Yin. Balance.

Sound like one of those New Age concepts? Perhaps.

Let me tell you 2 stories first…

Distractions and Newton’s 3rd Law

Garion was learning to control his power. Before this, it had always been spontaneous. He just thought it, and it’s manifested in reality. But there had always been some impetus, some urgency, to which he was forced to perform those acts of manifestation.

Now, he’s mindfully controlling what could be done. And his task at hand? To overturn a large piece of rock.

The first thing he noticed was, while he can concentrate on visualising the overturning of the rock, small peripheral happenings around him kept distracting him. A bird’s song. The buzz of a bee. The smell of the flowers. The coolness of the breeze. The light from the sun.

Finally he managed to ignore all the distractions, and was thinking about how to overturn the rock. Maybe he could lift it at one end? That made sense.

So he bent his will to lifting the rock at one end. Sweat beaded his brow. His hands were shaking, as though he had been physically lifting the rock.

Finally, Garion collapsed onto the soft grass in exhaustion. The rock simply would not move! After resting for some time, he got up and tried again. This time, he had a plan. Instead of slowly lifting, he would mentally grab hold of one end of the rock and flip it with one mental heave.

He visualised holding the end closest to him, and then he heaved. The rock flew off into the distance. Garion smiled in satisfaction.

Garion also realised he’s waist deep in the ground.

He didn’t brace himself against the impact. “To every action there is an equal and opposite reaction.” Flipping the rock upwards meant he was pushed downwards.

[this short story is paraphrased from the Belgariad by David Eddings]

The One Finger Punch

Chen Min was intrigued. How had that frail studious man defeated that ferocious brute of a man with just a touch of his finger? Chen Min had to find out.

“The essence of the skill,” the academic explained, “is balance.”

Suppose the total force was 10. If the attacker used 7 points of force, one just needed to use 3 points of force. If the attacker went all out, one merely needed a touch of a finger.

Matching the attacker’s force of 7 with another 7 would upset the balance. Sure one might still come out the better. But one had used an excess of energy to do it.

The academic, even though he’s not trained in kung fu, had agreed to teach Chen Min the skill. He took Chen Min to a spot beside a waterfall. It was noisy. The water splashed into the deep end of the pool. The leaves were rustling in the mildly strong wind.

And he told Chen Min to listen for birds. “What birds?” And 3 birds flew from a nearby tree.

Then he brought out a bird from the cage he carried and handed it to Chen Min. His instructions? Without holding onto the bird, with his palm open and the bird standing on his open palm, stop the bird from flying away.

Needless to say, the bird flew off without giving Chen Min any time to react.

Then the academic took another bird out from the cage. He held it in his hands. Chen Min could practically see the bird bending just a fraction of an inch downwards, in preparation for taking off.

And just as the bird was pushing off with its feet, the academic lowered his hands slightly, just enough to counter the force from the bird’s downward movement. Without the force needed to push off, the bird couldn’t fly off, and it stayed. The academic smiled, and lifted his hand and the bird flew away. Chen Min’s jaw dropped.

Chen Min stayed at the waterfall place, sat down, and concentrated. He focussed all his senses. He was watching intently at the trees. He was listening intently at the trees, desperately trying to ignore the rush of the wind and rustling of the leaves. He still couldn’t make out the chirps of the birds.

He thought maybe there weren’t any to begin with. And a few birds promptly took off from the trees into the sky. “I can’t do this!” and he slumped onto the ground.

Chen Min was enjoying the breeze, and was drifting into day dreaming mode, when he heard a chirp. He jerked back up, and tried to listen again. The chirping was gone.

He tried to understand what was going on. He was lying down, enjoying the breeze, not thinking about anything… and he was totally relaxed!

He sat back up with his legs crossed in meditative position, and slowed his breathing. And relaxed. And the faint chirping could be heard.

The key to maintaining balance in the One Finger Punch wasn’t to focus intently with one’s senses. It was to let go.

[that’s Chen Min, a kung fu master depicted in a comic book. Yes I get inspirations from comic books.]

Final words

Those 2 stories are part of what shaped my thoughts about self-control. Now, clench your fist. Then relax your fist. Now imagine your hand being in a clenched state and in a relaxed open state at the same time.

That’s what my ebook is about. To be disciplined enough to do the task you set out, and be relaxed enough to gently deflect interruptions. Hard and soft. Yang and Yin. Balance. Self-control.

I am also compiling a list of tips on handling interruptions of the I-want-to-focus-right-now-without-interruptions kind. This will complement the ebook, which will talk about deeper concepts (and so takes more time and effort to execute. Nobody said this was easy…). The compiled list will be freely available as a download for everyone.

If you want to contribute a tip on how you handle interruptions during your work, your studies, when you’re coding, anything, just put in a comment or contact me. I will include your tip in the list with attributions to you. Or you can tell me which awesome programmer I should be totally talking to, and asking that person for tips.

And finally, buy my ebook when it’s out. *smile*

Disclaimer: I’m not versed in New Age, Zen, Buddhism, and so on. I am not a health professional. I have some interest in those topics, only insofar as curiosity puts me. I’m just a simple man who happens to read a lot on a variety of topics.

[image by Neustock]

“The numbers don’t tally!” – a serial counting problem

Boy in shock

How many numbers are there from 7 to 26 (both inclusive)? How do you calculate your age?

Both solutions require you to count from one number to another. And if you’re quick-witted, you might have deduced that a subtraction shortens the process considerably. However, be careful of how you subtract.

For the first question, if you take 26 – 7, you get 19. But there are 20 numbers:
7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26

For the second question, you simply take the current year minus the year you were born in (let’s leave the exact months out of this). So I’m born in 1977, and as of this writing, it’s the year 2009, so I’m 2009 – 1977 = 32 years old. Which is correct. Wait, did I just tell you my age?

The army equipment

I’m going to relate to you an incident which happened when I was enlisted in the army. No it’s not a war story. It is, surprisingly, a counting problem.

I was a lowly private then, assigned to the store to help. On a particular day, the lance corporal I was helping was making sure of equipment stock. Basically ensuring that the stock number of a piece of equipment on paper, physically reflects the stock number of that piece of equipment in the store.

We took down all the (communications, I think) equipment down from the racks, and placed them neatly on the floor. Since the racks were now empty, we might as well clean them (army efficiency…). After that, the lance corporal and I started to count the equipment on hand.

It was a while later, and I was counting my part of the equipment, when the lance corporal swore. He was pacing, and gesticulating, and his face was contorting in expressions of worry I’ve never seen before.

“You ‘A’ level right? The numbers don’t tally! Tell me what’s wrong!” *

The lance corporal had counted the pieces of equipment. It tallied with what was recorded on paper. Then he sorted them by serial number (there was one on each piece of equipment). Then he matched them with the serial numbers recorded in the system. It was correct too.

And because the serial numbers were in increasing order, differing by 1, he did the subtraction trick. And found to his horror of horrors that it wasn’t the number he counted! Hence his panic.

Let’s say the serial numbers were:
SERIAL0007
SERIAL0008
SERIAL0009

SERIAL0024
SERIAL0025
SERIAL0026

The records said there were 20 pieces of this equipment on hand. But the subtraction gave him:
26 – 7 = 19!

What went wrong?

The problem was, the lance corporal didn’t count the equipment with serial number SERIAL0007.

The age problem

Let’s look at the age problem again. Say there’s a baby born in the year 2000. Which year would the baby be 1 year old? 2001. Which year would the baby be 5 years old? 2005.

How is the age calculated? Take the current year minus the birth year.

It works, because the birth year is not counted.

Back to the serial numbers

In the case of the serial numbers, each serial number had to be counted in.

Serial numbers SERIAL0007 and SERIAL0008 means there were 2 pieces of equipment, even though
8 – 7 = 1

Thus, the number of pieces of equipment for serial numbers SERIAL0007 through to SERIAL0026 should be 26 – 7 (+ 1) = 20

Conclusion

A series of numbers is easy to count the number of its members. Just use subtraction.

Just be careful to note whether the start of the series have to be counted.

=====
* In Singapore, ‘A’ level refers to the GCE ‘A’ levels, commonly taken by students at around 18 years old. If one passes this, one can proceed to the university (generally speaking). And in Singapore, having a degree means a lot.

And young men with an ‘A’ level certificate entering the army may sometimes be viewed or referred to with a slight derogatory attitude, albeit lighthearted and with a fun undertone. And with their status sometimes pronounced as “air level”.

[image by Izmabel]

Satellite and maritime data calls

Let’s talk maritime communication data. That’s what my users work hip-deep in.

“Wait! Didn’t you say you’re in the satellite business the last time?”

Oh yeah… so, to me (and my users), they’re effectively the same thing. You see, out at sea (ooh, alliteration), captains and sailors still need to talk with their loved ones back on terra firma. And surf the Internet (for you know, uh stuff…). And send (short message service) SMSs to their friends (and sweet phone messages to lovers). And report back to headquarters of course.

[actually they probably don’t frivolously send love notes. Because their company is paying for it, and paying close attention to their usage. See below…]

They do that by using a device, which sends data up to space to one of the supporting satellites (not our moon, one of those man-made ones), which then sends it back to Earth to be received by a satellite dish, which then relays that data out the normal way. Here’s a simple diagram to show how it works:

Basic land-sea data transfer via satellite

I totally suck at drawing diagrams…

The ships are equipped with a terminal (no it’s not installed at the fore of the ship like in the diagram. It’s probably in the command centre or communication centre or… you know, I have no idea where they install it…), which contains one (possibly more) subscriber identity module card or SIM card. These SIM cards aren’t the kind we use in our mobile phones though.

And each SIM card has one, possibly more, mobile numbers. And each mobile number is tied to a specific service, such as voice call, or SMS, or fax. These mobile numbers are used to track usage. What do I mean by that?

For the purposes of our discussion, the word “call” refers to any initiated communication. So an SMS is a call. Browsing to a particular website is a call. Faxing a piece of paper is a call.

And each call comes with associated data, and the entire thing is referred to as a call detail record, or CDR (or call record in short). There’s a lot of data in one call, so I’m going to tell you about the ones my users and customers care most about (because they track usage religiously. I know, because I wrote the web application for it.) You may encounter other similar terms.

  • Calling number – the mobile number making the call
  • Called number – the name’s a bit misleading. It can actually be an IP address or URL, other than a receiver’s mobile number or fax number. It’s also known as destination number.
  • Unit type – “messages” for SMS, bits/bytes/KB/MB (and other variants) for IP data, sec/min for voice data
  • Call duration – a value corresponding to the related unit type, such as 2.13 MB or 27 seconds
  • Call start date/time – when the call was made

The calling number is the unique identifier. The main thing it identifies is the company to whom the billed is sent (very important, this billing thing). The calling (mobile) number is tied to a billing account in the billing system.

The call duration is especially important, because the customer is billed accordingly. This is why I’m adamant about the use of fixed point database types such as numeric(12,4). The floating point errors will scale out of hand otherwise (millions of these call records are created every month). Which will cause the total billed amount to the customers to be out of hand. Money is a very serious topic… Do not underestimate the power of 2 numbers not tallying by a difference of 2 cents in the billing report…

That’s the basics of satellite data. Oh right, the terminals…

There’s this service called Broadband Global Area Network, or BGAN for short (pronounced bee-gan), and I even presented at a product launch hosted by my employer. The BGAN service is provided by Inmarsat, a satellite telecommunications company, and they provide me with most of the data I work with. And they (and their data servers) are based in London. Which is why I obsess over timezones and formatting dates and times.

So (customer) companies buy these BGAN terminals and install them on their ships. And the captains and crew members get to communicate with people on land, and even with other ships. All made possible by routing data through the satellites.

And my users sell these terminals, associated paraphernalia (such as the SIM cards), and data usage (where price plans come in). Inmarsat deals with telecommunications companies (like my employer), and the telecommunications companies take care of the customers.

There are other types of terminals of course. Which means different services, and different types of data. Which gives my team many headaches. Because my users will be telling the customers “We support this!”, and turn to my team to actually support the new data, new price plans, new billing calculations, new entries in bills, new everything. There are now more than 10 sources of call records which we handle every day…

I believe that’s enough information for now. Any questions?