Open XML SDK class structure

March 27, 2012 By

I’ve gotten a few questions on the class structure of the Open XML SDK. There are articles on Open XML itself, where you work directly with XML files and tags, and zip them up yourself. Basically you work with WordprocessingML (Word), SpreadsheetML (Excel), PresentationML (PowerPoint) and possibly DrawingML (for images and stuff). Eric White did a lot of stuff on this.

There are also articles on the use of Open XML SDK itself. However, the articles I’ve found tend to give you code samples and some explanation of why you do those things, but didn’t really explain the deep “why”.

The fundamental question I was asked was “How are the Open XML SDK classes related to each other?“. A related question is “Why do I have to use that particular class?”.

While I’m familiar with the spreadsheet portion of the SDK, I believe the general class structure applies to the WordprocessingML and PresentationML parts too. So I’ll use the SpreadsheetML part as the example.

I also didn’t find any article giving names to the class categories I’m going to tell you, so I’m going to make up my own. If there’s an official Microsoft article on this, let me know. Generally speaking, there are 4 types of SDK classes:

  • Relationship Part Classes (henceforth referred to as RPCs)
  • Root Classes (henceforth referred to as RCs)
  • Content Classes (henceforth referred to as CCs)
  • Special Classes

Before I continue, keep in mind that Open XML documents are basically a bunch of XML files zipped together. Just like a relational database, certain XML files are related to each other (just like certain database tables are related to each other). Which brings us to the first type of class.

Relationship Part Classes

RPCs are the glue that holds certain SDK classes together. They are most easily recognisable by their class type name. For example, WorkbookPart and WorksheetPart.

However, not all SDK classes with names that end with “Part” are RPCs. For example, TablePart is not an RPC (it’s actually a Content Class). The corresponding RPC is actually TableDefinitionPart.

The most important part of an RPC is that it carries a relationship ID, and this relationship ID is used to tie relevant classes together. An RPC is also different in that it has as a property, a Root Class.

Root Classes

Remember that Open XML documents are a bunch of XML files zipped together? Well, an RC represents one of those XML files.

For example, the RPC WorksheetPart has as its RC, the Worksheet class. The Worksheet class holds information that basically translates into an XML file, typically sheet1.xml (and sheet2.xml and so on). The Worksheet class contains your worksheet cell data information.

Content Classes

If RCs represent an XML file, then CCs are basically XML tags.

For example, the Worksheet class contains the SheetData class, which contains the Row class(es), which in turn contains the Cell class(es). The corresponding XML tags are “worksheet”, “sheetData”, “row” and “c”.

Yes, an RC represents an XML file, and also translates to be the first XML tag of that XML file. That’s why it’s called a Root Class, because it also represents the root element (of the underlying XML structure/document).

Special Classes

These aren’t really that special. As far as I know, there are only 3 classes under this category: WordprocessingDocument, SpreadsheetDocument and PresentationDocument.

Those 3 classes form the starting point of any code relying on the Open XML SDK. You can consider them as Super Relationship Part Classes, because their properties are mainly RPCs.

An illustration

You might still be confused at this point (I don’t blame you…). Here’s a diagram for a simple Open XML spreadsheet:
Open XML SDK class structure

In green, we have the Special Class SpreadsheetDocument as the ultimate root.

In blue, we have the RPCs, 1 WorkbookPart class and 2 WorksheetPart classes. The SpreadsheetDocument class has the WorkbookPart class as a property. The WorkbookPart class contains a collection of WorksheetPart classes.

In grey, we have the RCs, 1 Workbook class and 2 Worksheet classes. The Workbook class is the RC of WorkbookPart class. The Worksheet classes are RCs of corresponding WorksheetPart classes. The Workbook class represents the workbook.xml file and the Worksheet classes (typically) represent sheet1.xml and sheet2.xml files.

In orange, we have the CCs. The Workbook class contains the Sheets class, which in turn contains 2 Sheet classes. The Sheet classes have a property holding the relationship ID of the corresponding WorksheetPart classes, which is how they’re tied to the Worksheet classes.

One of the most confusing parts…

After working with the Open XML SDK for a while, you might find yourself asking these questions:

  • Why are there so many classes?
  • Why are some of these classes devoid of any meaningful functionality?
  • Why are some of these classes duplicates of each other?

When I was first using the SDK, I felt the same way when I first used the .NET Framework: Being overwhelmed. There were many namespaces, with many classes in them, and I didn’t know which class to use for a specific purpose until I looked it up and wrote a small test program for it. Having a comprehensive help database/file for the .NET Framework was a really good idea.

And so it was with the Open XML SDK. I mean, it’s a spreadsheet. I can see a couple dozen of classes. Maybe. It turns out to be a lot of classes. That’s why there are so many code samples out there, but you don’t really know why you need to use that particular class.

And there are classes without any meaningful properties or functions. They inherit from a base Open XML class, and that’s it. For example, the Sheets class.

Then there are classes with identical properties. For example, the InlineString class, the SharedStringItem class and CommentText class. Or the Color, BackgroundColor, ForegroundColor and TabColor classes.

The answer is the same for all the above questions. The Open XML SDK is meant to abstract away the XML file structure.

There are duplicate classes because each class eventually translates into an XML tag (if it’s a CC or RC). XML requires a different tag for different purposes, hence the Open XML SDK has different classes. Even though the classes are identical in programming functionality, they become rendered as different XML tags.

There are classes without any seemingly meaningful properties or functions because their sole purpose is to have children. (Ooh Open XML SDK joke!) The Sheets class has as children, the Sheet classes. In XML, they’re correspondingly the “sheets” tag with “sheet” tags as children. The final XML tags have no XML attributes, hence the corresponding SDK classes also have no properties. Tada!

And finally, there are so many classes, because frankly speaking, you need one SDK class corresponding to each individually different XML tag. There are a lot of XML tags used in Open XML, hence so many classes. And that’s before we add in the Relationship Part Classes.

If you have any questions, leave them in a comment or contact me. And I’ll see if I can answer them in an article.

Filed Under: Programming Tagged With: , , , , ,

Launching SpreadsheetLight

January 10, 2012 By

I am excited to tell you that my spreadsheet software library is available!

SpreadsheetLight

For the initial launch version (I decided to go for version 1. Why do people launch with versions 0.8? I don’t know…), you get comprehensive support for styles, rich text formatting, 47 named cell styles, themes (either one of the 20 built-in themes or create-your-own). Well, like I said, comprehensive styling support.

There’s also the (hum-drum) support for merging cells and freezing panes. I actually explored how to split panes. I certainly wrote about it in my Open XML guide, but it turns out that there’s a tiny rectangle at the top-left corner that Microsoft Excel didn’t tell me about. The size of that tiny rectangle is dependent on the font you use, and even the screen resolution of your computer screen.

While I could add a function that allows you to just input the size in EMUs (English Metric Units), I decided that if I can’t do it well, I don’t want to do it. At least for the initial launch.

Row heights and column widths were also big time drains. It turns out that they’re also dependent on the font and screen resolution of your computer screen. I was trying to calculate the standard row heights and column widths for the fonts in the built-in themes, and I thought I had them. I wrote a program using SpreadsheetLight to generate spreadsheets with different minor fonts, and I wrote a program to read in those spreadsheets and get the “standard” row height and column width. I spent 3 hours collecting data.

Then on a whim, I switched my computer screen’s resolution from 120 DPI to 96 DPI (my eyesight’s not that good ok? I need bigger text…), and whoa! All that data doesn’t apply anymore… All in all, I think I spent 6 or 7 days trying to figure out a general calculation formula. I failed. I don’t know how Excel does it.

I also surprised myself by including extensive support for pictures. I thought you just insert a picture into the worksheet and that’s it! It turns out there’s a ton of post-insertion manipulations you can do. For example, if your picture has transparent areas, you could set a background fill, and that background colour will be visible through the picture. Also, you can rotate the picture in 3D.

SpreadsheetLight is licensed under the MIT license. I decided to use one of the available software licenses instead of making up one of my own. As far as I can tell, the MIT license allows the recipient of the software to use the software in personal or commercial products. It’s also categorised as free software, as in freedom of use, not free as in cost. I don’t want to deal with per-client, or per-server, or per-developer or per-what-not licensing restrictions.

Even if you’re not interested in spreadsheet software, have a look at SpreadsheetLight. Tell some other programmer about it. Tell your manager about it. It took me slightly over 2 months of intense coding, and I want someone in the world out there to have an easier life because of SpreadsheetLight. Thanks!

Oh, and the image art is designed by Charlie Pabst from Charfish Design. While I have a fair competence in image work, I decided to get a professional designer to help me. It’s a business and professional product. I’m not going to risk the product’s success so I could stoke my ego…

Filed Under: Programming Tagged With: , , ,

Working on spreadsheet software library

December 5, 2011 By

As a natural and logical extension from my Open XML spreadsheet guide, I’m writing a software library to create and manipulate Open XML spreadsheets. (Never mind that decompiler project I was working on… 2 months of coding… sunk cost… moving on…).

I did some research (ok, an inordinate amount of research…) on the available spreadsheet software libraries out there, both free and commercial, both supporting Open XML (or .xlsx in any case) and the old .xls (Microsoft Excel in binary). I have 2 observations.

First, there’s a plethora of classes in the library. It’s sort of expected. There’s support for a lot of functionality, and it just burgeoned into many classes. Personally, I hate it when I have to learn a new library. There’s a whole bunch of documentation and classes I have to read up on and experiment to just do a simple thing (printing a string of characters is the first thing I try). When I first encountered the .NET Framework, I was crushed. It’s redeeming feature was its extensive documentation, which made learning easier.

Second, even though there’s support for a lot of functionality, it still takes quite a bit of code to accomplish what you want done (granted, much less than if you wrote low level code). Hey I wrote a guide on Open XML spreadsheets, I know how many lines of code needed to just create an empty Excel file, ok?

But these are spreadsheet software libraries!. They’re supposed to make your life easier. In fact, much easier.

I read that when the iPhone was designed, the engineers told Steve Jobs that it needed to have 4 or 5 buttons. Steve Jobs said no. One button (to rule them all). The iPhone now only has the 1 button.

So I took inspiration from that and designed my library to have that quality. Alright, alright, here’s a code sample:

SLDocument sl = new SLDocument();
sl.Save();

That will save an empty Excel file named “Book1.xlsx”. What, not Hello World enough for you?

SLDocument sl = new SLDocument();
sl.SaveAs("HelloWorld.xlsx");

There. Now the file is named “HelloWorld.xlsx”. What, sheet name? Most (if not all) of the libraries I researched required you to add a new worksheet to an empty file. All spreadsheets have at least one worksheet. Why force the programmer to do it anyway? You don’t see Microsoft Excel forcing the user to add worksheets in a newly created spreadsheet file, right? (Excel even has 3 worksheets added by default).

Alright, fine. The first worksheet’s name is by default “Sheet1″. You can rename it.

SLDocument sl = new SLDocument();
sl.RenameWorksheet(sl.DefaultFirstSheetName, "Hello");
sl.SaveAs("HelloWorld.xlsx");

There, happy? So, how do we set cell values?

SLDocument sl = new SLDocument();
sl.RenameWorksheet(sl.DefaultFirstSheetName, "Hello");
sl.SetCellValue(2, 3, 3.14159);
sl.SetCellValue(2, 4, "This is PI");
sl.SaveAs("HelloWorld.xlsx");

The cell with row 2, column 3 will have the value of PI. The cell with row 2, column 4 will have the string “This is PI”. Yes, the library supports cell references such as “C2″ and “D2″. My opinion? They make better sense to a user with visual interface to the spreadsheet. It’s much harder to use when you’re programming with a non-visual interface to the spreadsheet. Good luck iterating through rows 2 to 500,000, with columns 1 to 1000 (financial reports, I’m looking at you…).

Want to add a new worksheet?

SLDocument sl = new SLDocument();
sl.RenameWorksheet(sl.DefaultFirstSheetName, "Hello");
sl.SetCellValue(2, 3, 3.14159);
sl.SetCellValue(2, 4, "This is PI");
sl.AddWorksheet("SecondWorksheet");
sl.SetCellValue(5, 5, "Why am I not first?");
sl.SaveAs("HelloWorld.xlsx");

Hey, a software library is supposed to make your life easy. The second worksheet’s name is *drum roll*, “SecondWorksheet”. The string “Why am I not first?” is in row 5, column 5 of the newly added worksheet. How does the library know which worksheet to add which cell value? By magic. Ok, fine, it automatically keeps track of worksheets.

When a user enters a cell value in Excel, does the user need to know which worksheet? No, because that information is implied. The user knows which worksheet because the user chose it already. And so does this software library.

Oh yeah, I even have basic theme support!

SLDocument sl = new SLDocument(SLThemeTypeValues.Flow);
sl.RenameWorksheet(sl.DefaultFirstSheetName, "Hello");
sl.SetCellValue(2, 3, 3.14159);
sl.SetCellValue(2, 4, "This is PI");
sl.SaveAs("HelloWorld.xlsx");

That gives you the Flow theme, one of the built-in themes in Microsoft Excel (note: only the fonts and font colours are supported). You can even design your own custom theme.

System.Drawing.Color[] clrs = new System.Drawing.Color[12];
clrs[0] = System.Drawing.Color.White;
clrs[1] = System.Drawing.Color.Black;
clrs[2] = System.Drawing.Color.WhiteSmoke;
clrs[3] = System.Drawing.Color.DarkSlateGray;
clrs[4] = System.Drawing.Color.DarkRed;
clrs[5] = System.Drawing.Color.OrangeRed;
clrs[6] = System.Drawing.Color.DarkGoldenrod;
clrs[7] = System.Drawing.Color.DarkOliveGreen;
clrs[8] = System.Drawing.Color.Navy;
clrs[9] = System.Drawing.Color.Indigo;
clrs[10] = System.Drawing.Color.SkyBlue;
clrs[11] = System.Drawing.Color.MediumPurple;

SLDocument sl = new SLDocument("ColourWheel", "Castellar", "Harrington", clrs);
sl.RenameWorksheet(sl.DefaultFirstSheetName, "Hello");
sl.SetCellValue(2, 3, 3.14159);
sl.SetCellValue(2, 4, "This is PI");
sl.SaveAs("HelloWorld.xlsx");

There are 12 colours you need to define. These correspond to the 2 light colours, 2 dark colours, 6 accent colours, the hyperlink colour and the followed hyperlink colour. “ColourWheel” is the theme name, “Castellar” is the major Latin font and “Harrington” is the minor Latin font. The major Latin font is used when you apply the Title named cell style. The minor Latin font is basically the body font.

What named cell style? A customer suggested supporting the feature.

Named cell styles

So how do you apply it?

sl.ApplyNamedCellStyle(2, 3, SLNamedCellStyleValues.Good);
sl.ApplyNamedCellStyle(2, 4, SLNamedCellStyleValues.Accent1);

I am finishing up version 1 of the library, and it will soon be available. I’m targeting a launch in January 2012. The software library will be called SpreadsheetLight. The primary idea is for it to be simple and clean. Simple for you to use, and you write clean code when you use it.

My internal tests show that SpreadsheetLight runs faster than 2 other free libraries. I won’t tell you which 2, because it’s not relevant, and because I’m automatically biased, and because it’s just not nice to the other programmers who contributed to those 2 libraries. The point is that it runs fast and is effective, which I’m happy about.

Filed Under: Programming Tagged With: , , , , ,

First look at XML Studio

November 10, 2011 By

I’ve been working with XML files for a while (if you’ve been reading my blog for the past few months, you’re probably sick of the XML-related stuff…). Specifically with Open XML. While I don’t always read and write XML files, I do refer to the Open XML ECMA-376 documentation and the Open XML SDK help file a lot. And then, I go look at some XML files, just to check that I wrote them correctly.

I recently found out about XML Studio from Liquid Technologies. Disclaimer: I was contacted by a company representative, and given a free developer license for the software. But go check out their software if you’re doing XML-related stuff.

I blazed through the list of features and benefits, and settled on one. Oh my fishballnoodles they can generate C# source code from XML files! It uses the XML Data Binder.

So my first thought was: Can I use it to somehow generate source code that’s (sort-of) compatible with Open XML SDK?

Short answer: No. But that’s because the XML files don’t have the Open XML SDK class names in them, so you can’t really have source code working with the SDK.

However, my next thought was: Can I at least generate an XML file that would have been generated by the equivalent source code using the SDK?

First, I loaded an XML file from an Open XML spreadsheet (after renaming .xlsx to .zip and then unzipping and then get one of ‘em darn sheet.xml files). Then I found out that I couldn’t generate source code from this. *sad*

But I found out I could generate an XML schema from the XML file. Ohhkayy… Then I found out that generating source code required an XSD, an XDR or a DTD file. Alright, getting there.

Then I thought I could create a worksheet with some typical data so that I could grab the resulting XML file with some of the possible data types, which I could then use to generate a corresponding XSD schema file, and then generate corresponding source code. Note the recursive problem solving ability of my programmer mind.

And then it hit me that I could just use the correct schema file from ECMA-376. So I went to the second edition of ECMA-376 (latest is third edition as of this writing but not currently super-supported yet), and went to folder of part 1 (there are parts 1 to 4). Which has this very descriptive name of “ECMA-376, Second Edition, Part 1 – Fundamentals And Markup Language Reference”. Under this folder, there’s a zip file called “OfficeOpenXML-XMLSchema-Strict.zip”. And in that zip file is the motherlode of your schema dreams.

And so I opened up the schema file related to the Worksheet class of the SDK (which is sml.xsd). And got this:
XML Studio Schema View

Click image for larger view.

I’ve expanded the node for the Cell class. That’s awesome. You see that “0..1″ to the left of CT_CellFormula? That means a Cell class can contain 0 to 1 of the complex type (see “CT_” prefix) CellFormula. For nodes that take in at least 1 to an unlimited number of children nodes, you get “1..*”. This is reflected in the schema as minOccurs=”0″ for “you don’t really need it” and maxOccurs=”unbounded” for “you just have as many children as you want, ok? But make sure they’re of this type.”

As of this writing, I still haven’t managed to generate source code that does what I want (after a few hours of wheedling code around). But essentially, I’m trying to create an alternate Open XML SDK just from the schema information from ECMA-376. I’m pretty sure XML Studio wasn’t created for this in mind… I’ll keep you posted on my findings. If you have any XML editing stuff you think I should know, tell me, because I want to see if I can break, uh, I mean utilise XML Studio to its full potential.

Filed Under: Programming Tagged With: , , , ,

Decompiling Open XML spreadsheets

November 3, 2011 By

Ok, I’m going to reveal the big secret project that I’ve been working on for the last 2 months. I’m writing a software program that will decompile Open XML spreadsheets into C# and VB.NET source code.

Now I know what you’re thinking. “But Vincent, there’s that SDK Productivity Tool that does that already!”

Frankly, when I started the project, I didn’t even think about the SDK tool. But, when I looked at the generated source code from the SDK tool, I found it… hideous. There were 2 things I found annoying:

  • New classes were created willy-nilly
  • Properties were dumped into class instantiation using object initialisers

The first point meant that most of the classes were created one-off. It didn’t matter if you needed a class of type SomeClass multiple times. The SDK tool simply created another class of type SomeClass. If that class type was used multiple times, you’ll see variables named someClass1, someClass2 all the way to someClass21. It’s why I wrote about multiple use variables versus multiple variables.

The second point meant that if a class has many properties, you might end up with something like:

CellFormat cellFormat3 = new CellFormat(){ NumberFormatId = (UInt32Value)0U, FontId = (UInt32Value)10U, FillId = (UInt32Value)9U, BorderId = (UInt32Value)0U, ApplyNumberFormat = false, ApplyBorder = false, ApplyAlignment = false, ApplyProtection = false };

That’s one line of code.

The problem I have with object initialisers is when you need to comment something in between. Commenting in C# and VB.NET means an entire line is commented, although C# offers the /* comment */ variant. There’s just no easy way to do so. Compare with this:

cellFormat = new CellFormat();
cellFormat.NumberFormatId = 0U;
cellFormat.FontId = 11U;
cellFormat.FillId = 10U;
cellFormat.BorderId = 0U;
cellFormat.ApplyNumberFormat = false;
cellFormat.ApplyBorder = false;
cellFormat.ApplyAlignment = false;
cellFormat.ApplyProtection = false;

I just find that easier to pick and choose stuff I don’t want.

Now the big advantage (my differentiation or unique selling proposition) is that I offer VB.NET too. The SDK tool doesn’t. Here’s a snippet:

run = New Run()
run.RunProperties = New RunProperties()
run.RunProperties.Append(New FontSize() With {.Val = 11R})
clr = New Color()
clr.Theme = 1UI
run.RunProperties.Append(clr)
run.RunProperties.Append(New RunFont() With {.Val = "Calibri"})
run.RunProperties.Append(New FontFamily() With {.Val = 2})
run.RunProperties.Append(New FontScheme() With {.Val = FontSchemeValues.Minor})

You will notice that I do use object initialisers. “That’s hypocritical of you!”. Perhaps, but I use them when the number of properties is small. I’ve kept it to 3 for now. Object initialisers in my case also made it easier that I don’t have to declare and instantiate new classes with actual variable names.

I understand why the SDK tool generates source code the way it does. It has to do with completely iterating through every single part and class of the root class SpreadsheetDocument. If you’ve ever written code to traverse a tree structure, you’ll know how tedious it can be.

The one thing the SDK tool lacks about the source code it generates is context. It runs through the entire Open XML document structure like a squirrel looking for every single acorn on a tree. It doesn’t stop to check any acorn for size, defects or even if it’s an acorn. Look, winter’s coming soon, and the squirrel doesn’t have all day telling you that this particular acorn is related to that particular acorn, and no it doesn’t care how big the acorn is, it’s got the teeth to eat it, ok?

Why are we talking about squirrels again?

So, after about 20 thousand lines of code, I’m just barely getting my software into beta mode. Halfway through that, my heart sank with the enormity of the task. In order to generate more readable code, I cannot iterate through the XML tree structure like the SDK tool. I had to stop and make sense of what the class was.

That made me look at the SDK help file and the ECMA-376 specification file way too much… Did you know the ECMA spec is like over 5000 pages long? And that’s part 1. Parts 2, 3 and 4 are smaller, but still heavyweights in their own right. And there are so many classes and child classes and grandchildren classes and properties and…

I’m going to at least make a valiant effort to have the software self-complete on a subset of the Excel functionality (and thus a subset of the SDK). If you’re interested, I present to you SoxDecompiler. As of this writing, I’m just trying to see if people are interested in the software, so it’s just a page to collect email addresses of the people interested in the software. I think I wrote “interested” way too many times…

For some reason, the name conjures an image of a thread slowly unravelling a sock. But I like it. It stands for “Spreadsheet Open XML Decompiler”.

Filed Under: Programming Tagged With: , , ,
« Older Posts