“Entries cannot be opened multiple times in Update mode” error in SpreadsheetLight

So I finally got to work on upgrading SpreadsheetLight to .NET Standard 2. The main code changes revolve around the System.Drawing assembly reference. Changing the reference to the NuGet System.Drawing.Common package solved most of it. Maybe I’ll tell you about the handful of System.Drawing related changes in another article.

But the error that gave me minor heart attacks was this one:
“Entries cannot be opened multiple times in Update mode”

Going through the code, it’s the point where SpreadsheetLight saves the whole spreadsheet, and then opens it up again for minor “enhancement” fixes. These enhancements were to change the internal relation links in the file to be how Excel writes them, rather than what the Open XML standard uses.

It’s not wrong, it’s just that I want to align whatever SpreadsheetLight creates to be as close to what Excel outputs as possible.

The idea is that a spreadsheet that Excel outputs works on LibreOffice Calc and on Apple Mac software devices such as iPhone and iPad and Mac Books.

Apparently what the Open XML SDK outputs as a default conforms to the Open XML Standard, but doesn’t quite work in iPhone/iPad (the spreadsheet doesn’t open properly in iPhone/iPad).

So as the simplest of fixes go, I just commented that entire chunk of code. Making sure the SpreadsheetLight outputs spreadsheets that will work in Excel on Microsoft Windows is the bare minimum criteria. Everything else is an extra benefit.

SpreadsheetLight version 3.5

So I finally got SpreadsheetLight onto the .NET Standard/Core train. Get yours here.

The reason it took me so long is that I was still recovering from personal and business stuff. And when I saw I needed to set up Visual Studio 2019 (2017 back then), and I had to check if my code was compatible with .NET Standard/Core, that was too much…

Another reason was whether I should still depend on the Open XML SDK DLL, or just bite the bullet and depend solely on the official Microsoft DocumentFormat.OpenXml Nuget package.

In the end, based on emails sent to me, and observation of general software development trends, I just went ahead and depend on Nuget packages. Because the urgency and cost of me continuing to stall was just too great.

The main code incompatibility, so to speak, has to do with the absence of System.Drawing in the .NET Standard framework. SpreadsheetLight uses the Drawing assembly to check text widths and heights for the purposes of autofitting columns and rows. I didn’t want to have to find an alternative method…

Until I found that Microsoft has an official Nuget package System.Drawing.Common which mimics (or is exactly?) the System.Drawing but in .NET Standard/Core form. Woobdeedoo!

Yes, I’m like 2 years too late. I might possibly have lost developers who jumped ship and used other libraries. I have emails of developers/users asking for a .NET Standard/Core version, which means they are kind of forced to keep their applications running on .NET Framework 4 because of me (probably high opinion of myself, but a sense of self-importance drove me to action, so maybe it’s not that bad if it’s not publicly known. Oh wait, I just told you…)

So get a copy of the latest and greatest version of SpreadsheetLight and I’ll talk to you another time.

Rapid iterations and long running programs

So recently I have to run a program that takes a long time to complete. I don’t mean the program is slow. It’s just that whatever it’s doing takes a long time because there is a lot of data to process.

I’m talking days, stretching into almost 2 weeks.

When I was younger (and more naive), I thought pfft there could never be programs that had to run more than an hour. Computers are so fast!

I was wrong.

When I was writing my honours thesis in university, I had to write a computer virus infection simulation program (it forms the basis of my thesis results). I had nodes serving as computers. I put edges in the graph (mathematical term for network) as connections between the “computers”. Then I let my virus logic run amok in the network.

When I found out how long it took to complete, or at least reach a stable infection equilibrium, I was shocked. Dismayed. Dread even. It took slightly over an hour to complete. Sometimes two hours.

So I’d run the program and go for lunch. Then come back and see what my results were. I learnt to make changes and note them down meticulously because re-running the program was expensive, time-wise.

And so it is with my current long-running program. One mistake, or one forgotten consideration, or one miswritten line of code, and I’d have to rerun the program. By that time, I might have wasted a day.

As they say, measure twice cut once.

Python pass by reference

Some time ago, I was given the opportunity to take a free course in Python. I was like “Sure why not? And it’s free? Woot!” (I don’t actually say “woot” but it seems like the kind of thing people say, right?)

The course is a beginner module meant for technical people (but not full-on coders). I was the only professional programmer in the class. I didn’t care, I don’t know Python. So 2 points of interest…

White space is critical

The programming languages I know consider white space as something to make the code easier to read for the human. Python elevates that to a whole new level.

I’m not even talking about the debate about whether a tab or 4 spaces is better for indentation. Python will punish you if you used 4 spaces for indentation here and then use 3 spaces for indentation there. This is especially terrible if you use an editor where a tab looks like 4 spaces so it looks like the indentation is fine but it’s not.

Parameter values are by default passed by reference

This one took a bit of getting used to. I’m used to passing values to functions and knowing that those values (or classes or structs) will never be tampered with unless I say so. For example, with the “ref” keyword in C#:

public void Foobar(ref ComplicatedClass ImComplex)
{
// ImComplex will possibly be modified
}

I’ll just slide that into my “good to know, but I’ll probably never use” list. I don’t want any weird errors slithering up because that’s hard to swallow. (I swear that’s the end of my snake jokes)

P.S. Happy Independence Day to you if you celebrate it. Coincidentally, it’s also the official day that my company closed. So Independence Day to me too.

How to fix Open XML SDK created spreadsheet for iOS devices

Note that this is only for Open XML SDK 2.0. I haven’t tested for SDK 2.5 (or later, but there’s no “later” version at the point of this writing).

The fix

You have to manipulate the [Content_Types].xml file. This XML file is zipped together in the spreadsheet file, and is at the root of the file. If you don’t know what I’m talking about, you’re probably reading the wrong article.

Unfortunately, this can’t be done within Open XML SDK. As I understand it, the SDK runs internally on System.IO.Packaging namespace objects.

Now a package is a zip file with structure. But a zip file is not necessarily a package. (go maths logic!) See Microsoft reference (bottom of page).

As I understand it, you can’t manipulate a package’s [Content_Types].xml with any of the methods or classes from the System.IO.Packaging namespace. This XML file is supposed to be tamper-proof.

So if you can’t change it with the Packaging stuff, then you’ll have to manipulate it as a pure zip file. And as at .NET Framework 3.5 (because I’m using Open XML SDK 2.0 as reference), there’s no in-built zipping mechanism for the kind of zipping algorithm you need (the Gzip and Deflate isn’t exactly the tool).

This means you need an outside zip library. I leave it to you to find your favourite library. Assuming you found one, here’s why you need to change the [Content_Types].xml file.

The one generated by Excel looks something like this:

<?xml version="1.0" encoding="UTF-8" standalone="true"?>
<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">
<Default ContentType="application/vnd.openxmlformats-package.relationships+xml" Extension="rels"/>
<Default ContentType="application/xml" Extension="xml"/>
<Override ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml" PartName="/xl/workbook.xml"/>
<Override ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml" PartName="/xl/worksheets/sheet.xml"/>
<Override ContentType="application/vnd.openxmlformats-officedocument.theme+xml" PartName="/xl/theme/theme.xml"/>
<Override ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.styles+xml" PartName="/xl/styles.xml"/>
<Override ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sharedStrings+xml" PartName="/xl/sharedStrings.xml"/>
<Override ContentType="application/vnd.openxmlformats-package.core-properties+xml" PartName="/docProps/core.xml"/>
</Types>

The one generated by Open XML SDK 2.0 looks something like:

<?xml version="1.0" encoding="UTF-8"?>
<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">
<Default ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml" Extension="xml"/>
<Default ContentType="application/vnd.openxmlformats-package.relationships+xml" Extension="rels"/>
<Override ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml" PartName="/xl/worksheets/sheet.xml"/>
<Override ContentType="application/vnd.openxmlformats-officedocument.theme+xml" PartName="/xl/theme/theme.xml"/>
<Override ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.styles+xml" PartName="/xl/styles.xml"/>
<Override ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sharedStrings+xml" PartName="/xl/sharedStrings.xml"/>
<Override ContentType="application/vnd.openxmlformats-package.core-properties+xml" PartName="/docProps/core.xml"/>
</Types>

Spot the difference time!

The workbook part is missing. Specifically, look for the XML tag with the ContentType attribute equal to “application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml”

Open XML SDK, for whatever reason, doesn’t explicitly set the PartName=”/xl/workbook.xml” part. This is why iOS devices can’t see the spreadsheet file.

I don’t know whether the fix will work on Android devices or Windows phones. But a developer emailed me that iOS 6.0 devices seem to still be able to read the spreadsheet files by Open XML SDK, but iOS 7.0 just gives up.

Microsoft, please, you created the Open XML specs. Other companies are following the specs and the one useful tool you made to help generate Open XML documents fail to follow the specs.

Considerations for storing Excel cell value in code

You may hate Excel, but you may find a discussion of how Excel stores cell values interesting.

So I have a spreadsheet library. The biggest concern at the initial stage was how to store all the spreadsheet data efficiently. I hear people talking about millions of cells, so I’m scared. If my program stores a spreadsheet cell using 10 bytes (for example), a million cells would take up 10 million bytes in memory.

Let’s start by looking at all the different types of information you can type into a spreadsheet cell. You have:

  • booleans: TRUE or FALSE
  • numbers
  • text
  • rich text (different styled text within the entire text itself)
  • dates and times

For us programmers, “numbers” can be separated into floating point or integer types. An Excel user won’t see a difference.

So how does Excel actually store those values? I’m going to focus only on Open XML because I’m not interested in BIFF files…

  • booleans: TRUE stored as text “1” and FALSE stored as text “0”
  • numbers: stored as text
  • text: duh
  • rich text: stored in a separate shared strings list, with the index to that list stored as text here.
  • dates and times: stored as number that’s in text form

You will see everything is basically stored as text. That’s because the underlying XML files are text files. There’s a property (XML attribute) that differentiates the data, such as boolean, number, string, inline string, shared string.

So why are dates stored as a number? It’s easier to do date calculations with 41449 than “24 June 2013”. So how is this number obtained? See here.

So if you’ve been looking closely enough, Excel’s optimisation tactic is to store everything as numeric text as far as possible. So I want to follow that.

Before doing so however, I went to read what other people are doing AKA open source spreadsheet libraries. In code, they use an object to store the cell value. As in System.Object, the mother of all data types in .NET.

So you have an integer? Dump it into the object variable. Floating point? Dump into object. String of characters? Dump.

How do you read it out? Boxing and unboxing. You remember it’s a floating point value and cast it back from an object to a double variable type.

So what did I do? I have a double variable and a string variable, and I store the cell value in one or the other based on the input.

The “all in object” way has variable (no pun intended) memory size, based on the contents. Sort of. I’m not an expert in this.

My way has a fixed memory size for double’s. Each double takes up 8 bytes (for sure?). A string variable takes up variable size, but because the optimisation tactic is to store data as a number, I can assign the data to the double variable and set the string variable to null. This means the string variable size is sort of fixed too.

So this is what I do. If it’s a number, I store it in the double variable and set the string variable to null. If it’s text, I convert it to a number by using shared strings (out of scope for discussion here) and store the index into the double variable and set the string variable to null. The only cases where the string variable is actually used is if I store the text there, or if I want to store the actual number there (because “1.23456789” may not be stored exactly as that in a double variable. Go read on how floating points are implemented for details), which are rare.

According to Jon Skeet, strings take up 20 + (n/2)*4 bytes (where n is the number of characters). But a null string takes up 8 bytes (it’s either 4 or 8 bytes. I’ll assume the worse scenario).

This means for the most part, each cell has a double variable that takes up 8 bytes and a null string that takes up 8 bytes. A cell value of 10 or 3.14 or 12345678.9 takes up 16 bytes regardless.

Since 16 bytes is less than 20 + (n/2)*4 bytes, I save more memory in most cases. I also have less boxing and unboxing operations, which make things go faster.

File upload size limit in IIS

Yay file uploads. As if letting the users to type in stuff into the web application giving me SQL injection nightmares weren’t enough, now I have to let users upload files.

Peachy.

So during my investigations into the limits of file uploading, I found that I couldn’t upload a file more than 30MB on my test server. It failed faster than Superman could jump a building in a single bound, and with just as much sound.

In short, here are my findings. The default file size limit set in IIS (6 and below? Read on for more details) is 4MB. In IIS7 (on Windows Server 2008), the file size limit is 30MB (technically it’s 28.61MB because it’s 30000000 bytes but who’s keeping track. Hey you read on!).

So how do you change the limits? In the web.config file. We’re doing ASP.NET applications.

<httpRuntime executionTimeout="3600" maxRequestLength="20480" />

That will give you a timeout period of 1 hour (3600 seconds) and a file size limit of 20MB (20480 KB. Yes, that attribute is in kilobytes).

For IIS7, we do this:

<security>
    <requestFiltering>
        <requestLimits maxAllowedContentLength="134217728" />
    </requestFiltering>
</security>

That gives you a 128MB limit (128 * 1024 * 1024). Yes it’s in number of bytes.

So why was I doing file uploads? Documents from university staff or students. The most important of which is the final doctoral thesis.

I asked how large can that thesis be, assuming it’s in PDF form. I got an answer where a 40MB limit seems too small. Really?

I had trouble auto-generating an Excel file of 40MB just to test the server limits. Do you know how large 40MB is?

If it’s a video or sound file, then yes I can believe it. I have video files of over 100MB, some over 200MB. But a PDF? With mostly text?

Go check out the Open XML specs from ECMA. The largest document is about 28MB. It’s over 5000 pages. I doubt any thesis can match that number of pages.

Specious spreadsheet security

If not for Chinese, it would’ve worked.

So in Excel, you can set a password for either protecting a particular worksheet, or protecting the entire workbook/spreadsheet. This password is then hashed, and the result is stored within the spreadsheet contents.

Now with the use of Open XML spreadsheets, this means the resulting hash is stored in “plain text” within XML files. Without going into too much detail, here’s the algorithm for the hash as documented in the Open XML SDK 2.0 help docs:

// Function Input:
//    szPassword: NULL-terminated C-style string
//    cchPassword: The number of characters in szPassword (not including the NULL terminator)
WORD GetPasswordHash(const CHAR *szPassword, int cchPassword) {
      WORD wPasswordHash;
      const CHAR *pch;
 
      wPasswordHash = 0;
 
      if (cchPassword > 0)
            {
            pch = &szPassword[cchPassword];
            while (pch-- != szPassword)
                  {
                  wPasswordHash = ((wPasswordHash >> 14) & 0x01) | ((wPasswordHash << 1) & 0x7fff);
                  wPasswordHash ^= *pch;
                  }
            wPasswordHash ^= (0x8000 | ('N' << 8) | 'K');
            }
      
      return(wPasswordHash);
}

This algorithm is wrong. Or at least it doesn't give the resulting hash that Excel produces.

Granted, I didn't really expect the algorithm to be correct. Because from the SDK help:

An example algorithm to hash the user input into the value stored is as follows:

It's an example algorithm, so it may or may not be the one that Excel actually use. However, for the purposes of usability, the password used by users have to be encrypted using Excel's algorithm, so we have to somehow get the resulting hash. Either we get the algorithm itself, or we simulate an algorithm such that the resulting hash matches that of Excel's.

Which is what Kohei Yoshida did. See his modified algorithm. This algorithm worked!

Given that I'm Chinese, I did what's natural: I used Chinese characters as the password.

And the modified algorithm failed. It only worked if the password consisted only of Latin alphabets. I tried Japanese characters. Failed too.

This is why I don't support password protection in SpreadsheetLight. I don't want to give the false impression that the worksheet/workbook encryption works. This is one feature where "partially worked" is unacceptable.

Granted Open XML spreadsheets also support other types of encryption, and you can store the name of the algorithm and salt value you used, and even the number of times the hash algorithm was run.

But.

This is in the context of a spreadsheet library.

What are spreadsheet libraries mostly used for? Automation.

Which means minimal (if at all) human interaction and intervention.

And so the question comes up.

Where do you store the password?

If a human is protecting the worksheet/workbook, she will provide the password herself, and then encrypts it (well Excel does it), and she just remembers the password.

If a spreadsheet library is generating the workbook, and encrypting it, the password has to be gotten from somewhere, right? So it's stored, maybe in a text file, maybe in a database, maybe hardcoded in code, or whatever.

The point being that the password is not held solely by a human being. And a computer hard drive is easier to hack into than a human brain.

And I will prefer not to be a party to facilitating insecure spreadsheet generation. Besides, it's Open XML. The data is supposed to be open and sharable. Password protection seems to be the opposite. Even Microsoft cautions the use of depending just on encrypted Excel files.

We already have social engineering used by devious people to deceive people into giving their passwords over. Storing passwords on a computer seems suicidal because computers have no common sense at all.

As a final word, I'd say using the Open XML SDK can be either verbose, or obscenely painstakingly verbose. Simple tasks need a couple of dozens of lines of code, and complex tasks take at least 2 magnitudes of work to do. For individualistic, compartmentalisable (that's not a word, right?) tasks, you can do it from scratch. Add even a smidgeon of complexity, and you'll find a library more useful. Try mine.

SpreadsheetLight version 3

Version 3 of my spreadsheet library is now available. There’s a whole bunch of updates, including Excel 2010 conditional formatting such as data bars with negative value fill colours and icon sets with no icons.

SpreadsheetLight is possibly the most developer-friendly spreadsheet library ever. Even if I do say so myself. 🙂

Modulo 26 and column names

I was sitting in the lecture theatre valiantly trying to keep awake. The professor was speaking on the rigorous application and proving of the modulus function. It’s basically the remainder, but I’ve never been introduced to it in such, uh, rigor.

He brought up an example using modulo 26. And demonstrated the wrapping around of values. And the use of it in cryptology (a class I took later on, and I got tasked by the cryptography professor to write a program to do simple encryption. But another story perhaps…).

Modulo 26 is similar to finding the remainder. The difference is that the remainder is unique. This is important to our discussion.

“About what?” you may ask.

Excel column names.

There are 2 types of cell references used in spreadsheets, the R1C1 format and the A1 format. The R1C1 is simple. If the row index is 5 and the column index is 7, the result is R5C7.

The A1 format takes on a column name and the row index. Using our example again, the result is G5, because “G” is the 7th alphabet. Yes, that list of 26 alphabets.

The version of Excel currently (Excel 2007 and 2010) has up to 3 letters, with XFD as the last column name (that’s the 16384th column). What happens is you have column names A, B, and then up to Z. Then the next column name is AA, then AB and then up AZ. Then BA, BB and so on.

Basically, it’s base 26 arithmetic.

As far as I know, the typical method of getting the column name given the column index, is to run a loop. You add 1 until it hits 26, then you move to the next “position”, and then start from 1 again.

There’s nothing wrong with this method. It’s just that you have to iterate as many times as the given column index. If you’re given 16384, the loop runs 16384 times. This is regardless of the fact that the result is always the same. Given the range of values, the result can only be one of 16384 values.

So it was with this in mind, and my dabbling in game development (which said “Precalculate everything!”), that I precalculated an array of strings that are the column names. The array has 16384 items. The context is a spreadsheet software library.

Now to recoup that precalculation cost, I’d have to access my array at least 16384 times. This is where the context of the spreadsheet library comes in. Everybody (and their dog) wants to know if my library can handle millions of cells. This means if the column name calculation is in a method, that method is called millions of times. Given the iteration loop in it, that means the method with the iteration loop thing isn’t efficient (it’s O(n^2)).

However, due to technical issues, I can’t keep the array of strings. The column name array needs to be static to be available throughout the library. This causes issues if multi-threads or multi-processors or multi-whatevers comes in.

So I can’t use my static array anymore. Bummer. The O(n) of simple array access was working so well.

But I still want to have an efficient way of getting column names. So instead of iterating, I used simple division and modulus operations.

Consider 587. How do you know there’s 5 hundred? 587 / 100 equals 5 (truncating remainders). How do you know there’s 8 tens? 87 / 10 equals 8 (truncating remainders).

Yes, it’s elementary arithmetic, but it works great. So we can do the same for column names. There’s a problem though.

In the case above, we divided by 100. Why 100? Because it’s 10^2, and we’re concerned with the 2nd position after the ones position. The ones position is 10^0 by the way, which is 1.

So for our case, for the “hundreds” position, we divide by 676, which is 26^2. And for the “tens” position, we divide by 26.

Now 587 is 100*5 + 10*8 + 7. I’m going to use the notation (5,8,7) to denote this.

Now consider a column index of 3380. 3380 is equal to 676*5 + 26*0 + 0. This is (5,0,0).

However, in our case, our acceptable range of values is 1 through 26. It doesn’t contain zero. So (5,0,0) is not valid.

In the case of 587, we’re working in base 10, with the acceptable range of values being 0 to 9. This is “proper” remainder. Given any number in base 10, there’s a unique number within [0, 9] that’s the remainder.

However, for our purposes, there’s no unique number. Because we’re working in modulo 26, not just “remainder 26”.

The correct column name corresponding to column index 3380 is “DYZ”. This corresponds to (4,25,26). Or
3380 = 676*4 +26*25 + 26.

Note that 3380 is also 676*5 + 26*0 + 0.

My solution is to start from the “ones” position. If it’s greater than zero, fine. If it’s less than or equal to zero, borrow from the next larger position. Then we move to the next larger position, and check again. Continue to borrow until there are no zero values (or negatives) on the “right” side of the resulting notation (we can have “leading” zeroes).

So (5,0,0) becomes (5, -1, 0 + 26), or just (5,-1,26), borrowing 1 from the “tens” position. We cannot have -1, so that becomes (4, -1 + 26, 26), which becomes (4, 25, 26).

An interesting effect is that we typically assign 0 to A, 1 to B, and 25 to Z. In this case, 1 is assigned to A, 2 is to B, and most interestingly, both 0 and 26 map to Z. In fact, any multiple of 26 will map to Z.

Don’t think Z is special. Any multiple of 26 plus 1 also maps to A. So 1, 27, 53 and so on map to A. This is a property of the modulo thing.

Do you have a better way of converting (5,0,0) to (4,25,26)? Let me know in the comments.