Friday, November 16, 2007

AJAX.Net & The Freelance Project...

I've been steadily plugging away at my freelance project that I've been working on for way too long now. Luckily, my partner/customer in this endeavor is quite understanding of the time involved in caring for young children and life in general. Luckily, I've managed to make a number of updates to the back-end processing that makes front end processing much easier. Basically, there are a number of things where I was doing some rather heavy calculations for every call, that I didn't need to be making.

For example, the state of a element. This state changes every time more elements are added into the collection, so I can't just give an element a single state when I load that element and be done with it. At the same time, a lot of calls are taken up with calculating that particular state at other times during the life-cycle of the process. Well, my solution was to have a table in the database which records the element and its current state, and create a stored procedure which can be ran to recalculate those states. Also, I'll soon be sending that data down to the page code as a JSON variable so that I won't need to make some of the calls to the web server while playing with elements not in the initial data set.

Basically, a foo object can have 36 elements, but approximately only 18-25 of those elements will come down with the initial retrieval of foo attributes. It's all part of the processing to determine what is displayed that way. Anyways, if I want to include one of those elements that aren't in that initial set in my output, the current process is that I tell the web page, it builds the div for displaying the element, and then sends off a request to the web service backend to get its state. Sure, I'm not doing that a whole lot, but if I can kill a call to the web service, that makes the process just that much faster, because we don't have to wait on the round trip time.

Of course, now that I've got my backend updated, I'm facing some odd issues with the front end. The front end is heavy in JS and CSS because, well it just is. I have the Prototype and libraries in play, and then it is also a AJAX.Net site. This means it's spitting out a whole bunch of JS. Anyways, my issue is that when I load the site from the testing web server, it's not displaying the elements. Other attributes are changing, but the elements aren't. Oddly, this behavior isn't being replicated on my development box.

I'm not sure if it's an AJAX.Net issue or a Prototype issue. Or something else altogether. Somewhat annoying if you ask me.

Thursday, November 8, 2007

Speeding up File Writing

I'm dealing with rather large data sets these days. I'm talking on the order of a few million records easy. One of the tasks I've been given is to take these records and generate a flat file of them to transfer to another process for external processing.

Not a problem, right? In fact it could be done just from the Query Analyzer with its ability to save results to a file.

Well, sometimes that works all right, but this is a process that needs to be run routinely, plus at random other times of the year. So I shoved it into a simple console application where I have a configuration file, and I can override those configuration options via command line arguments.

It worked beautifully for my first few test sets of data I ran through it, the largest being about 5,000 records. Then I pulled the full test data set, which has about a million records. After staring at that black screen for what felt like forever, I went about hunting a way to speed things up.

My studies uncovered the fact that IO.Stream classes were faster than Microsoft.VisualBasic.FilePut classes. The problem there was that I was already using the IO.Stream classes.

So, I stumbled through things, hunting for ways to improve my efficiency. I tried IO.File.CreateText as a way to return a StreamWriter, but that had no discernable efficiency bonuses (added an extra 300-400 milliseconds at 1,000 records).

I tried using a StringBuilder instance to build the file and then shoved it into the StreamWriter all at one time (as opposed to writing directly to the file every line the way I had been doing). Oddly, that still had no discernable efficiency bonuses (added an extra 100-200 milliseconds at 1,000 records).

Almost ready to give up, I went looking through the IO.File class to see what functions I could use. Which is when I stumbled upon IO.File.AppendText. Giving an internal shrug, I set about reworking my function to use a StringBuilder class and the AppendText file.

As an aside, note that if the file exists, I always delete it so that I'm creating a new file every time.

Imagine my surprise when I saw a rather marked difference in efficiency, as evidenced by a two-second drop in speed. Not quite sure what to make of it, I ran the tests again, and then increased my records to 5,000 and then to 100,000.

I was flabbergasted at these results:
1,000 Records

Method 1:
1. 0:0:16:47
2. 0:0:14:344
3. 0:0:14:406

Method 2:
1. 0:0:12:687
2. 0:0:12:78
3. 0:0:12:343
5,000 Records
Method 1:
1. 0:0:25:0
2. 0:0:24:686
3. 0:0:24:702

Method 2:
1. 0:0:12:280
2. 0:0:12:124
3. 0:0:12:265
100,000 Records
Method 1:
1. 0:4:23:249
2. 0:4:25:798
3. 0:4:25:675
Method 2:
1. 0:0:18:952
2. 0:0:21:77
3. 0:0:20:983
I couldn't believe it. The AppendText function working with a StringBuilder class was nearly 4 minutes faster than writing the file directly. Sure, it means I have to touch the data an extra time, but with that as a speed increase, that is something I'm quite willing to live with.

Monday, November 5, 2007

Merging Data Tables....

I found myself in a situation where I was loading a data table with data, and then possibly bringing in additional data. I knew that .NET 1.1 the ability to merge whole datasets was included, but little did I know how they upgraded that in .NET 2.0.

I was playing with the underlying table of data, and found the .MERGE command. Surprised, I did a quick Google search on it, and found this article describing all the changes to the ADO.NET DataTable class.

Now, if they had only included an APPEND capability, I'd have been even happier.

Blog Widget by LinkWithin