Using the Search Sitemap Provider with the ASP.NET Futures release

Sitemaps

First off a quick introduction to what a Sitemap is, according to http://www.sitemaps.org...

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site.

ASP.NET Futures Release

The ASP.NET futures release can be found at http://www.asp.net/downloads/futures/. Just a note, as you will see on the futures site, the content in the futures release might not make it to the official ASP.NET Ajax release. The futures release contains functionality that the team is thinking about incorporating into the main ASP.NET release. So this means that you could play with the code and hope that the team incorporates it into ASP.NET Ajax.  Enough with the disclaimer.

Download the ASP.NET Futures  (July 2007) Release

Getting Started Search Sitemap Provider

The ASP.NET team created the SearchSitemapProvider as part of the ASP.NET Futures (July 2007) release. The release contains a HttpHandler to process the calls, SearchSiteMaps.axd, which serves up the Sitemap and two providers, AspNetSiteMapSearchSiteMapProvider and DynamicDataSearchSiteMapProvider. The AspNetSiteMapSearchSiteMapProvider will generate a static Sitemap based on the ASP.NET sitemap. The DynamicDataSearchSiteMap provider provides a way to dynamically generate the Sitemap.

To enable Search Sitemaps in your ASP.NET you need to do the following.

• Edit the web.config to add the searchSiteMap section

 1: <microsoft.web.preview>

 2: <searchSiteMap enabled="true">

 3: <providers>

 4: <add name="Navigation" type="Microsoft.Web.Preview.Search.AspNetSiteMapSearchSiteMapProvider, Microsoft.Web.Preview"/>

 5: <add name="Articles" type="JosephGuadagno.Web.BLL.SiteMap.ArticleSiteMapData, JosephGuadagno.Web.BLL" 

 6: targetUrl="" 

 7: targetUrlseparator="/" 

 8: queryStringDataFormatString="{0}" 

 9: queryStringDataFields="Url"/>

 10: <add name="Books" type="JosephGuadagno.Web.BLL.SiteMap.BookSiteMapData, JosephGuadagno.Web.BLL" 

 11: targetUrl="" 

 12: targetUrlseparator="/" 

 13: queryStringDataFormatString="{0}" 

 14: queryStringDataFields="Url"/>

 15: <add name="News" type="JosephGuadagno.Web.BLL.SiteMap.NewsSiteMapData, JosephGuadagno.Web.BLL" 

 16: targetUrl="" 

 17: targetUrlseparator="/" 

 18: queryStringDataFormatString="{0}" 

 19: queryStringDataFields="Url"/>

 20: </providers>

 21: </searchSiteMap>

 22: </microsoft.web.preview>

• Edit the web.config to add the httpHandlers section

 1: <httpHandlers>

 2: <add verb="*" path="SearchSiteMaps.axd" type="Microsoft.Web.Preview.Search.SearchSiteMapHandler" validate="true"/>

 3: </httpHandlers>

A call to http://www.josephguadagno.net/SearchSiteMaps.axd will produce something like this.

 1: <sitemapindex>

 2: <sitemap>

 3: <loc>

 4: http://www.josephguadagno.net//SearchSiteMaps.axd?sitemap=Navigation

 5: </loc>

 6: <lastmod>2007-11-13T04:53:21.371Z</lastmod>

 7: </sitemap>

 8: <sitemap>

 9: <loc>

 10: http://www.josephguadagno.net//SearchSiteMaps.axd?sitemap=Articles

 11: </loc>

 12: <lastmod>2007-11-13T04:53:21.371Z</lastmod>

 13: </sitemap>

 14: <sitemap>

 15: <loc>

 16: http://www.josephguadagno.net//SearchSiteMaps.axd?sitemap=Books

 17: </loc>

 18: <lastmod>2007-11-13T04:53:21.371Z</lastmod>

 19: </sitemap>

 20: <sitemap>

 21: <loc>

 22: http://www.josephguadagno.net//SearchSiteMaps.axd?sitemap=News

 23: </loc>

 24: <lastmod>2007-11-13T04:53:21.371Z</lastmod>

 25: </sitemap>

 26: </sitemapindex>

This will instruct the search engine that there are at least four sitemaps that are available.  Now the search engine will crawl these four URLs and get the additional URLs that are available. The four sitemaps point to the four providers that are in the web.config file.  The reason for four of them are that one of them is for the fixed navigation tied to the site map file, the other three use the dynamic site map provider, one for each type since the classes and data retrieval are different.

Using the Search Sitemap Provider

AspNetSiteMapSearchSiteMapProvider

This provider is the easiest to use and only requires a valid Asp.Net site map file. Adding the following line to the searchSiteMap section of the web config file will instruct the provider to load the default asp.net site map.

 1: <add name="Navigation" 

 2: type="Microsoft.Web.Preview.Search.AspNetSiteMapSearchSiteMapProvider, Microsoft.Web.Preview"/>

DynamicDataSearchSiteMapProvider

The DynamicDataSearchSiteMapProvider is used when you have dynamic content that you want to submit to the search engines.  Dynamic content is content that is generated on the fly or does not have a fixed URL. As you see in lines 5 - 19 of my searchSiteMap providers section, I use three different custom providers, Articles, Books and News.  These providers inherit from the DynamicDataSearchSiteMapProvider class which requires that you implement the DataQuery method which returns a IEnumerable interface. I choose to return a List<SiteEntry> objects.

The SiteEntry class supports the following properties (taken straight from the Asp.Net futures site):

• The targetUrl property (required) specifies the URL of the page in the sitemap.
• The targetUrlseparator property (optional) specifies the seperator between the URL and the data fields. The default is ?. You can specifiy characters such as # or /.
• The queryStringDataFormatString property (optional) specifies how the data fields are formatted using syntax like that used by the String.Format method. If no format is specified, the provider auto-generates a default format string.
• The queryStringDataFields property (optional) specifies which columns you want to bind in targetUrlFormatString. If the property is not specified, the provider infers the list of column names from the collection returned by the DataQuery method.
• The lastModifiedDataField property (optional) specifies the column that contains information about the last time the sitemap was modified. This date should be in W3C datetime format, which allows you to omit the time portion and provide the date in the format YYYY-MM-DD. If the property is not specified, the provider attempts to read a property named SiteMapLastModified.
• The changeFrequencyDataField property (optional) specifies how frequently the page is likely to change. Valid values are: always, hourly, daily, weekly, monthly, yearly, never. If the property is not specified, the provider attempts to read a property named SiteMapChangeFrequency.
• The priorityDataField property (optional) specifies the priority of this URL relative to other URLs on your site. Valid values range from 0.0 to 1.0. If the property is not specified, the provider attempts to read a property named SiteMapPriority.
• Set the pathInfoFormat property (optional) to true if you want to use only the value in the URL (for example, http://site/page.aspx/1)

These properties are used in conjunction with your properties that you want to display in the site map. For my search site map I use the following fields:

The Url, SiteMapLastModified, SiteMapChangeFrequency, SiteMapPriority properties are set in code and the targetUrl, targetUrlseparator, queryStrignDataFormatString and queryStringDataFields are set in the web config.

So what do all of there properties mean and do. Well like I mentioned above the DynamicDataSearchSiteMapProvider expects a DataQuery method to be implemented.  This method is expected to return an IEnumerable (collection) of objects. In my case, I return a List<SiteEntry> objects.  My SiteEntry class looks like this.

The ArticleSiteMapData provider looks like this

 1: /// <summary>

 2: /// A SiteMap provider for the Article page.

 3: /// </summary>

 4: public class ArticleSiteMapData: DynamicDataSearchSiteMapProvider 

 5: {

 6: public ArticleSiteMapData() {}

 7:  

 8: public override IEnumerable DataQuery()

 9: {

 10: // Holds a list of the ArticleEntry's

 11: List<SiteEntry> list = new List<SiteEntry>();

 12: TList<Article> articles = new TList<Article>();

 13:  

 14: articles = DataRepository.ArticleProvider.GetAll();

 15:  

 16: foreach (Article article in articles)

 17: {

 18: string url = BLL.URLRewrite.GetArticleRewrite(article);

 19: SiteEntry siteEntry = new SiteEntry(url, article.AddedOn, SiteEntry.ChangeFrequency.Daily, "0.5");

 20: list.Add(siteEntry);

 21: }

 22:  

 23: articles = null;

 24: return list.ToArray();

 25: }

 26: }

Lines 14 - 21 you would replace with your code get your dynamic data.  Lines 19 and 20 is where the SiteEntry object is created and added to the List.

So how does the List<SiteEntry> tie into the provider configuration?  The answer is in the attributes/properties that are defined for provider. For this sample we have targetUrl ="", targetUrlSeparator="/", queryStringDataFormatString="{0}" and queryStringDataFields="Url" this tells the provide to build a string with the base of targetUrl, then add targetUrlSeparator, followed by calling string.format using queryStringDataFormatString with the parameters equal to the value of the properties listed in queryStringDataFields.  In other words the url will look like this.

http://www.yourdomain.com/<value of url field>

In my site, I generate the Url in code so that it will be consistent between different pages.  A "real" world example might look like this.
Properties: targetUrl ="BookReviews", targetUrlSeparator="/", queryStringDataFormatString="{0}/{1}.aspx" and queryStringDataFields="Id, Title" which would generate a url similar to this.

http://www.yourdomain.com/1/title.aspx

Notice that I had two field in the queryStringDataFields property.  You can list as many as you want, they just need to be comma separated. This means that the SiteEntry object needs to have at least the two properties of Id and Title.

For more information

That's about it.  I was a lot to read and digest but hopefully useful. I you have any questions, please feel free to contact me or leave a comment or you can discuss the Sitemap provider in the ASP.net forums.

Search Engines were you can submit a feed

Here some search engine urls where you can submit you site maps.

Google: https://www.google.com/webmasters/tools/siteoverview

Yahoo: https://siteexplorer.search.yahoo.com

Programming .NET Components, 2nd Edition

• Introducing Component-Oriented Programming
• .NET Component-Oriented Programming Essentials
• Interface-Based Programming
• Lifecycle Management
• Versioning
• Events
• Asynchronous Calls
• Multithreading and Concurrency Management
• Serialization and Persistence
• Remoting
• Context and Interception
• Security
• Interface-Based Web Services
• Unifying Windows Forms and ASP.NET Security
• Reflection and Attributes
• Generics
• C# Coding Standard

Microsoft® SharePoint® 2007 Development Unleashed

• Use CAML, SharePoint’s XML-based language for defining content, manipulating searches, and more
• Work with the SharePoint Object Model
• Build reusable packages for easy deployment to SharePoint server farms
• Program SharePoint webs, sites, document libraries, and files
• Leverage SharePoint 2007’s improved lists and new list events
• Manipulate and query meetings and Meeting Workspaces
• Integrate external business data into SharePoint applications
• Construct business workflows for enterprise content management and other applications
• Program ASP.NET-based SharePoint Web Parts, from the basics to state-of-the-art techniques
• Create Web Parts that can provide and consume data through connections
• Use SharePoint 2007’s built-in Web services for managing document and Meeting Workspaces, imaging, and lists
• Centralize spreadsheet storage and management with Excel Services
• Manage user profiles and enhance application security
• Debug and deploy SharePoint 2007 applications
• Work with Records Repositories and metadata

Pro ASP.NET 2.0 in C# 2005

ASP.NET 2.0 Website Programming: Problem - Design - Solution (Programmer to Programmer)

• Account registration, personalization and theming
• News and events, organized into categories
• Opinion polls
• Newsletter
• Forums
• E-commerce store with shopping cart and order management
• Localization

• Full online back-end administrative section, to manage practically all data from an intuitive user interface
• Site deployment

• Master pages
• Theming
• Personalization & Web parts
• Membership & Profile modules
• Personalization
• The new server-side UI controls such as GridView, DetailsView, FormView, Wizard, MultiView, the new xxxDataSource and navigation controls, among others.
• The new compilation mode and deployment modes
• The new framework for instrumenting the site, as well as handling & logging exceptions
• The new ADO.NET 2.0 features (e.g. caching with database dependency)
• The new classes for easy distributed transactions Management