Single Query for Efficient Paging with Total Row Count

Sunday July 13thtsql Category

I will not go into detail about the use of row_number for paging as there are many articles about it. Most of the articles either build dynamic queries then exec it, which I think defeats the purpose of using stored procedures and is difficult to maintain, or create two stored procedures to page and the other to get the count.

Here’s an alternative using a single query:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
set nocount on
 
declare @computeTotalRows bit
declare @page int
declare @pageSize int
 
set @computeTotalRows = 1
set @page = 1
set @pageSize = 25
 
with Companies as (
    select 
        RowId = row_number() over(order by main.Name)
        , main.*
    from (
        -- main query here, not affected by inclusion of RowId
        select distinct c.*
        from Company c
        where IsActive = 1
    ) as main
)		
select *
    , TotalRows = case 
        when @computeTotalRows = 1 then (select count(*) from Companies)
        else -1
    end 
from Companies
where RowId between (@page - 1) * @pageSize + 1 and @page * @pageSize
order by Name

Separating the main query from the outer query allows for use of the distinct qualifier without being affected by the introduction of the RowId column. (distinct is not necessary in this trivial example) The trick to computing the total rows is reusing the table expression again: select count(*) from Companies. The query is easy to main since it is not duplicated nor dynamically built in a string and is efficient since TotalRows is only computed when the bit is set. I am making an educated guess that computing the total rows within the same query provides performance gains by reusing SQL’s buffers. (I’m not a DBA, I wear the hat.) The total rows should be computed on the first page of a fresh search.

Tertiary Operator Gets No Love

Tuesday July 8thcsharp Category

The tertiary operator has gotten a bad rap. Some code cop said avoid it. Let’s transform an array of anonymous objects to HTML with the help of FormatEx

Model.Links = new[] {
    new { Link='www.microsoft.com', Text='microsoft' },
    new { Link='www.google.com', Text='google' }
    new { Link='www.yahoo.com', Text='yahoo' },
    new { Link='asp.net', Text='asp.net' },
};

To

<a href='www.microsoft.com' class='one'>microsoft</a>
<a href='www.google.com' class='two'>google</a>
<a href='www.yahoo.com' class='dont-bluff-the-chip-leader'>yahoo</a>
<a href='asp.net' class='one'>asp.net</a>

To transform using the tertiary operator do something like this

<%= Model.Links.FormatEx(n => { return
    n % 3 == 0 ?  "<a href='${Link}' class='one'>${Text}</a>" :
    n % 3 == 1 ?  "<a href='${Link}' class='two'>${Text}</a>" :
                  "<a href='${Link}' class='dont-bluff-the-chip-leader'>${Text}</a>";
})%>

To transform the recommended way is this

<%= Model.Links.FormatEx(n => {
    if (n % 3 == 0)
        return "<a href='${Link}' class='one'>${Text}</a>";
    else if (n % 3 == 1)
        return "<a href='${Link}' class='two'>${Text}</a>";
    else
        return "<a href='${Link}' class='dont-bluff-the-chip-leader'>${Text}</a>";
})%>

While I agree the if-else feels more comfortable, is it because of habit? The tertiary form looks cleaner in simple cases. The tertiary operator is certainly no less readable than lambdas.

Custom C# Syntax for WP-Syntax

Monday July 7thwordpress Category

Here is my custom C# syntax file for WP-Syntax , the plugin used to color C# code on this site. Copy the expanded file to wp-content/plugins/wp-syntax/geshi/geshi/csharp.php. Peform a diff (I would) if you are worried about updating a PHP file. I added a few keywords and changed the colors to match the default scheme in Visual Studio 2008.

download custom csharp.php

WordPress rules!

Self-Scheduling Execution Pattern

Monday July 7thpowershell Category

Definition

The Self-Scheduling Execution Pattern allows a script to be executed interactively or executed via Task Scheduler when the script registers itself with the service.

Explanation

There are many scenarios that warrant periodic execution of a script. For example, a script to invoke my Data Caching Service. The creation, testing and scheduling of a script usually follow these steps:

  1. Create the script. Define the logic using Powershell statements.
  2. Test the script. Execute the script interactively. Verify and debug. If it doesn’t work as expected repeat Step 1.
  3. Schedule the script through Task Scheduler

During steps 1 and 2, the script needs to obviously be interactive. Once tested, the script is ready for scheduling. Scheduling the script is laborious and we can do better with Powershell. The algorithm of the pattern is simply:

1
2
3
4
5
6
if scheduling
    schedule this script
else if unscheduling
    unschedule this script
else
    do the work

Translated for the command line:

To schedule: ps> .\script.ps1 -schedule
To unschedule: ps> .\script.ps1 -unschedule
To execute once: ps> .\script.ps1

Script

The script is fairly simple. The script schedules a task with a unique name derived from the full path of the script. Note the use of the event log. Remember this is not interactive once scheduled. A log is a necessity. (Our network administrator prefers the event log.) This script is a template for any logic you want to schedule. Just fill in Do-Work and schedule it. Remember to adjust the /MO 120 to the desired minutes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
param ([switch] $schedule, [switch] $unschedule)
 
function Main {
    # derive unique taskname from full path of script
    $scriptName = $MyInvocation.ScriptName
    $taskname = $scriptName -replace '\W','_'
 
    if ($schedule) {
        schtasks /create  /sc MINUTE /MO 120 /tn "$taskname" /tr "powershell -noninteractive $scriptName" /ru "SYSTEM"
    }
    elseif ($unschedule) {
        schtasks /delete /tn "$taskname" /f
    }
    else {
        pushd (split-path -parent $scriptName)
        $message = Do-Work
        echo $message
        Write-EventLog $taskName $message
        popd
    }
}
 
function Write-EventLog($source, $message) {
    $evt = new-object System.Diagnostics.EventLog("Application")
    $evt.Source = "$source"
    $infoevent = [System.Diagnostics.EventLogEntryType]::Information
    $evt.WriteEntry("$message",$infoevent,70)
}
 
function Do-Work {
    # do some work
    return 'message'
}
 
Main

Example

Here’s an example of how to get the content of an HTML page to execute a remote Data Caching Service .

32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
function Do-Work {
    $url = "http://localhost/ironmvc/services/cron"
    $webClient = new-object System.Net.WebClient
    $webClient.Headers.Add("user-agent", "PowerShell Script")
 
    $startTime = get-date
    $output = $webClient.DownloadString($url)
    $endTime = get-date
 
    $status = 'FAIL'
    if ($output -like "success") {
        $status = 'SUCCESS'
    }
 
    return "$status`t`t" + $startTime.DateTime + "`t`t" + ($endTime - $startTime).TotalSeconds + " seconds"
}

Inspiration

“Powershell script that can schedule itself to run later”, Mike Wilbur

"Quick and Dirty Web Site Monitoring with PowerShell", Otto Helweg

Data Caching Service with MVC

Sunday July 6thaspnetmvc Category

Caching infrequently updated data is an easy way to improve the performance of your website. Consider the example of reading an RSS feed and displaying the items on a web page. In all likelihood the RSS feed is only updated every few hours or even every few days. With this in mind let us alter the the usual process flow from

user views page -> page reads rss xml from external site -> page renders items -> send page to user

to

user views page -> page reads xml feed from cached file -> page renders items -> send page to user

Instead of hitting an external site directly, read the RSS feed from a file cache. How to update the file cache is described later in this article. Here is an example of the default index action for a page with a list of RSS links.

1
2
3
4
5
6
7
8
9
public ActionResult Index() {
    var model = new HomeModel();
 
    // read the RSS items from a file and set property for the view
    model.RssItems = ReadRssFeedFromFile("gotmvcnet.xml");
 
    // pass model to strongly-typed view
    return View(model);
}

In the view, render a list item for each RSS item.

<ul><%= Model.RssItems.Take(4).FormatEx(
    "<li><a title='${Description}' href='${Link}'>${Text}</a></li>"
)%></ul>

Reading and parsing the RSS feed is easy with the open-source RSS.NET library:

1
2
3
4
5
6
7
8
9
10
11
12
13
IEnumerable ReadRssFeedFromFile(string fileName) {
    var filePath = Server.MapPath(@"~\App_Data\" + fileName);
 
    var feed = RssFeed.Read(filePath);
    var channel = feed.Channels[0];
    var items = from RssItem item in channel.Items
                select new LinkItem {
                    Text = item.Title,
                    Link = item.Link.ToString(),
                    Description = item.Description
                };
    return items;
}

After the above code your view should render as before except now it gets its feed from a file. Let us now create an ASP.NET MVC service to create or update the RSS cache file gotmvcnet.xml whenever a specific URL is hit on the website . I’ll name the action after the unix cron utility. The url to trigger the service will be: http://yoursite/services/cron . Create an action named Cron in your service controller. Within this action add the logic to update the RSS cache.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
public class ServicesController : Controller {
    public ActionResult Cron() {
        string result = "success";
        try {
            UpdateRssFeeds();
        }
        catch (Exception e) {
            _log.Error(e);
            // do not explain why to public
            result = "fail";
        }
 
        return new ContentResult {
            Content = result,
            ContentType = "text/plain",
        };
    }
 
    void UpdateRssFeeds() {
        SaveXml("http://gotmvc.net/feed", "gotmvcnet.xml");
    }
 
    // save xml feed to App_Data
    void SaveXml(string uri, string fileName) {
        var xml = XDocument.Load(uri);
        var fullPath = Server.MapPath(@"~\App_Data\" + fileName);
        xml.Save(fullPath);
    }
}

To update the cache, the cron URL has to be hit periodically. I use a Powershell script following the Self-Scheduling Execution Pattern template to hit the URL every couple of hours. Another option is to use wget to hit the page and manually schedule a job with schtasks or visually through Task Scheduler.

Many operations can be performed within the Cron action such as sending e-mails, caching lookup data from the DB, copying the log file to a central location, etc. If you are deploying to a web farm, remember to hit services/cron url on each server in the farm.

The local file system is more efficient than any database. There is no connection cost, no network latency. I have used and seen some fairly complicated data caching schemes. In the end, the local file system is the most efficient and least troublesome. Keep it simple.

JSON Services Are Easy

Saturday July 5thaspnetmvc Category

JSON services are ridiculously easy with ASP.NET MVC.

1
2
3
4
5
6
7
8
9
10
11
12
public ActionResult GetCities() {
    var query = this.ReadFromRequest("q");
    var zipRows = Locations.StartingWith(query);
    var rows =  from r in zipRows
                select new { 
                    City = r.CityName, 
                    State = r.StateId, 
                    Zip = r.ZipId 
                };
 
    return Json(rows);
}

Line 11 returns a JsonResult which actions the ASP.NET MVC framework to set the content type and serialize the data into JSON format. Note how projection at lines 5-9 may be used to rename the columns into names expected by the AJAX consumer.

That’s all there is to it. The beauty of this is zero configuration. The service is part of your site and works with ASP.NET MVC routing.

Coding With Graphic Designers In Mind

Saturday July 5thaspnetmvc Category

We’ve come along way from classic ASP days but it seems most of web development is still geared towards developers with little love for the creative web designer. A web designer skillset includes HTML, CSS and a little bit of programming. In our team designers are asked to fix things such as verbiage, links, and simple HTML because a good portion of time is spent integrating comps, html mock-ups and CSS.

Consider this snippet from Phil Haack’s Code Based Repeater article:

1
2
3
4
5
6
7
<table>
    <% Html.Repeater<Hobby>("Hobbies", "row", "row-alt", (hobby, css) => { %>
    <tr class="<%= css %>">
        <td><%= hobby.Title%></td>
    </tr>
    <% }); %>
</table>

From a developer’s point of view the snippet rocks. Some cool stuff there and there’s no code repetition. However, a designer will likely not understand it. A designer can pick out the table row and guess he needs to edit this section. The issue is what if a designer wants to tweak or add an element to an even cell? The code above does not allow someone without C# skills to safely edit it.

My approach is to use format strings that closely resembles the rendered output. <% %> fragment bloat is minimized by coding within short blocks. It looks even cleaner if you don’t mind the tertiary operator. (I hope I am not the only one bothered by all of the <% %> bloat in trivial MVC examples on the web. Most examples look untidy and I don’t want to be reminded of my classic ASP days.)

1
2
3
4
5
6
7
8
<table>
    <%= Model.Hobbies.FormatEx(n => {
        if (n % 2 == 0) 
            return "<tr class='row'><td>${Title}</td></tr>";
        else 
            return "<tr class='row-alt'><td>${Title}</td></tr>";
    })%>
</table>

where FormatEx is an extension method and ${Title} is a format specifier for the HtmlEncode value of current item’s Title property (!{Title} is for raw output)

A designer will likely not understand the Func<int, string> on line #2 nor the modulus operator on #3, but a designer can search for class='row' and fix tickets quickly. And yes there is repetition of the format string. Designers tend to grasp that easier than tricky code.

I am not a fan of all the methods in HtmlHelper class. They hide too much of the HTML markup. A foreach loop with HtmlHelper methods adds too much <% %> visual noise going in and out of inline code and markup. To that end, I created FormatEx to do most of the rendering in my views. I prefer raw HTML markup since I view HTML in Firebug and manipulate HTML with jQuery. I have been shackled too long by ASP.NET webform controls. The signal-to-noise ratio was horrible. Web development is fun again having bare metal code.

1
2
3
4
5
6
7
8
<ul>
    <%= Model.Hobbies.FormatEx(item => {
        if (item.Title == Model.CurrentHobby) 
            return "<li class='selected'>${Title}</li>";
        else 
            return "<li>${Title}</li>";
    })%>
</ul>

On a side note, when will C# have string interpolation? Developers do so much string manipulation and C# provides nothing much better than printf. Why not something like

"""<li>And he said, "hello ${name}"</li>"""

instead of

string.Format(@"<li>And he said, ""hello {0}""</li>", HtmlEncode(name))

Powershell + SSH Remoting

Saturday June 21stpowershell Category

Microsoft seems to be up to its old tricks. Powershell 2 CTP 2 remoting is not currently supported on Windows XP nor 2003 Server. Something happened from CTP 1 to CTP 2 and the official word is they will “try” to support 2003 Server in future releases. Our entire production environment is built on 2003 Servers.

One solution is to use SSH and SFTP. Unfortunately, most of the solutions I have read on the web do not work or are too expensive. I’m not against paying for licenses but some of the licensing schemes are ridiculous for technology that is free in the Linux world. Why not a one time charge instead of paying for each server or each CPU? Don’t be discouraged, there is a completely free, no nags solution that works with Powershell 1 and 2.

I’m assuming you have installed Powershell on the server correctly. In a nutshell here’s what you need:

  • freeSSHd SSH daemon server
  • PuTTY SSH client
  • VILE (optional) simple vi editor for remote editing

Powershell in PuTTY Terminal

There are several free SSH server options and freeSSHd is one of two that echos the powershell prompt. The other one was nagware. freeSSHd also has the smallest memory footprint around 7M. There are two noteworthy options in freeSSHd . The first is the console application to use; point to the powershell executable. On 2003 server, the location is C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe. The second option is authorizing users. Users must be added to authorize them to use SSH, Telnet and/or Tunneling. SSH is all that is needed. One caveat, freeSSHd does not respect user settings. If I log in as administrator, the starting directory is c:\windows\sytem32. $HOME is not set. An easy work-around is to “dot source” a script a configuration script on the remote server.

PuTTY is an excellent SSH client. You may want to try the other utilities on PuTTY’s site such as the SFTP command line utility.

I prefer FileZilla (S)FTP client for transferring and editing files. There is an option to set the text editor, which basically allows local editing of a file in your favorite editor by clicking view/edit. When the file is saved FileZilla detects any changes and prompts you to upload. If you know vi, try VILE. It was the only vi editor that worked through the terminal. I have used vim in the past on cygwin and for whatever reason vim doesn’t work quite so well in this terminal configuration.

I did not detail the steps as it was fairly simple to configure. However, leave a comment if I should create a post detailing the steps.

WordPress on ASP.NET

Saturday June 21stwordpress Category

My livelihood is centered around .NET, but that does not mean I turn a blind eye to other platforms. I believe in using the right tools for the job. When it comes to blogging and web content management in general, open-source and PHP cannot be beat. There is little point in trying to reinvent the wheel in this space, though many try (BlogEngine.NET, dasBlog, Graffiti CMS). WordPress is free, has a much larger community of users and contributors.

WordPress runs fine on ASP.NET. This site runs on IIS7 and powered by WordPress 2.5.1.

WordPress rules!

Size

Colors