Sitemaps
First off a quick introduction to what a Sitemap is, according to http://www.sitemaps.org...
Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site.
ASP.NET Futures Release
The ASP.NET futures release can be found at http://www.asp.net/downloads/futures/. Just a note, as you will see on the futures site, the content in the futures release might not make it to the official ASP.NET Ajax release. The futures release contains functionality that the team is thinking about incorporating into the main ASP.NET release. So this means that you could play with the code and hope that the team incorporates it into ASP.NET Ajax. Enough with the disclaimer.
Download the ASP.NET Futures (July 2007) Release
Getting Started Search Sitemap Provider
The ASP.NET team created the SearchSitemapProvider as part of the ASP.NET Futures (July 2007) release. The release contains a HttpHandler to process the calls, SearchSiteMaps.axd, which serves up the Sitemap and two providers, AspNetSiteMapSearchSiteMapProvider and DynamicDataSearchSiteMapProvider. The AspNetSiteMapSearchSiteMapProvider will generate a static Sitemap based on the ASP.NET sitemap. The DynamicDataSearchSiteMap provider provides a way to dynamically generate the Sitemap.
To enable Search Sitemaps in your ASP.NET you need to do the following.
- Edit the web.config to add the searchSiteMap section
1: <microsoft.web.preview>
2: <searchSiteMap enabled="true">
3: <providers>
4: <add name="Navigation" type="Microsoft.Web.Preview.Search.AspNetSiteMapSearchSiteMapProvider, Microsoft.Web.Preview"/>
5: <add name="Articles" type="JosephGuadagno.Web.BLL.SiteMap.ArticleSiteMapData, JosephGuadagno.Web.BLL"
6: targetUrl=""
7: targetUrlseparator="/"
8: queryStringDataFormatString="{0}"
9: queryStringDataFields="Url"/>
10: <add name="Books" type="JosephGuadagno.Web.BLL.SiteMap.BookSiteMapData, JosephGuadagno.Web.BLL"
11: targetUrl=""
12: targetUrlseparator="/"
13: queryStringDataFormatString="{0}"
14: queryStringDataFields="Url"/>
15: <add name="News" type="JosephGuadagno.Web.BLL.SiteMap.NewsSiteMapData, JosephGuadagno.Web.BLL"
16: targetUrl=""
17: targetUrlseparator="/"
18: queryStringDataFormatString="{0}"
19: queryStringDataFields="Url"/>
20: </providers>
21: </searchSiteMap>
22: </microsoft.web.preview>
- Edit the web.config to add the httpHandlers section
1: <httpHandlers>
2: <add verb="*" path="SearchSiteMaps.axd" type="Microsoft.Web.Preview.Search.SearchSiteMapHandler" validate="true"/>
3: </httpHandlers>
A call to http://www.josephguadagno.net/SearchSiteMaps.axd will produce something like this.
1: <sitemapindex>
2: <sitemap>
3: <loc>
4: http://www.josephguadagno.net//SearchSiteMaps.axd?sitemap=Navigation
5: </loc>
6: <lastmod>2007-11-13T04:53:21.371Z</lastmod>
7: </sitemap>
8: <sitemap>
9: <loc>
10: http://www.josephguadagno.net//SearchSiteMaps.axd?sitemap=Articles
11: </loc>
12: <lastmod>2007-11-13T04:53:21.371Z</lastmod>
13: </sitemap>
14: <sitemap>
15: <loc>
16: http://www.josephguadagno.net//SearchSiteMaps.axd?sitemap=Books
17: </loc>
18: <lastmod>2007-11-13T04:53:21.371Z</lastmod>
19: </sitemap>
20: <sitemap>
21: <loc>
22: http://www.josephguadagno.net//SearchSiteMaps.axd?sitemap=News
23: </loc>
24: <lastmod>2007-11-13T04:53:21.371Z</lastmod>
25: </sitemap>
26: </sitemapindex>
This will instruct the search engine that there are at least four sitemaps that are available. Now the search engine will crawl these four URLs and get the additional URLs that are available. The four sitemaps point to the four providers that are in the web.config file. The reason for four of them are that one of them is for the fixed navigation tied to the site map file, the other three use the dynamic site map provider, one for each type since the classes and data retrieval are different.
Using the Search Sitemap Provider
AspNetSiteMapSearchSiteMapProvider
This provider is the easiest to use and only requires a valid Asp.Net site map file. Adding the following line to the searchSiteMap section of the web config file will instruct the provider to load the default asp.net site map.
1: <add name="Navigation"
2: type="Microsoft.Web.Preview.Search.AspNetSiteMapSearchSiteMapProvider, Microsoft.Web.Preview"/>
DynamicDataSearchSiteMapProvider
The DynamicDataSearchSiteMapProvider is used when you have dynamic content that you want to submit to the search engines. Dynamic content is content that is generated on the fly or does not have a fixed URL. As you see in lines 5 - 19 of my searchSiteMap providers section, I use three different custom providers, Articles, Books and News. These providers inherit from the DynamicDataSearchSiteMapProvider class which requires that you implement the DataQuery method which returns a IEnumerable interface. I choose to return a List<SiteEntry> objects.
The SiteEntry class supports the following properties (taken straight from the Asp.Net futures site):
- The
targetUrl property (required) specifies the URL of the page in the sitemap.
- The
targetUrlseparator property (optional) specifies the seperator between the URL and the data fields. The default is ?. You can specifiy characters such as # or /.
- The
queryStringDataFormatString property (optional) specifies how the data fields are formatted using syntax like that used by the String.Format method. If no format is specified, the provider auto-generates a default format string.
- The
queryStringDataFields property (optional) specifies which columns you want to bind in targetUrlFormatString. If the property is not specified, the provider infers the list of column names from the collection returned by the DataQuery method.
- The
lastModifiedDataField property (optional) specifies the column that contains information about the last time the sitemap was modified. This date should be in W3C datetime format, which allows you to omit the time portion and provide the date in the format YYYY-MM-DD. If the property is not specified, the provider attempts to read a property named SiteMapLastModified.
- The
changeFrequencyDataField property (optional) specifies how frequently the page is likely to change. Valid values are: always, hourly, daily, weekly, monthly, yearly, never. If the property is not specified, the provider attempts to read a property named SiteMapChangeFrequency.
- The
priorityDataField property (optional) specifies the priority of this URL relative to other URLs on your site. Valid values range from 0.0 to 1.0. If the property is not specified, the provider attempts to read a property named SiteMapPriority.
- Set the
pathInfoFormat property (optional) to true if you want to use only the value in the URL (for example, http://site/page.aspx/1)
These properties are used in conjunction with your properties that you want to display in the site map. For my search site map I use the following fields:
| Url |
Custom Field |
| SiteMapLastModified |
specifies the last time the content changed |
| SiteMapChangeFrequency |
specifies how often this page is expected to change |
| SiteMapPriority |
specifies the priority of this Url verses others |
| targetUrl |
specifies the url that will be used |
| targetUrlseparator |
specifies the separator between the query string |
| queryStringDataFormatString |
specifies how the sitemap provider will format the url |
| queryStringDataFields |
specifies what properties from the SiteEntry object you will use to pass to the queryStringDataFormatString (similiar to the String.Format) |
The Url, SiteMapLastModified, SiteMapChangeFrequency, SiteMapPriority properties are set in code and the targetUrl, targetUrlseparator, queryStrignDataFormatString and queryStringDataFields are set in the web config.
So what do all of there properties mean and do. Well like I mentioned above the DynamicDataSearchSiteMapProvider expects a DataQuery method to be implemented. This method is expected to return an IEnumerable (collection) of objects. In my case, I return a List<SiteEntry> objects. My SiteEntry class looks like this.
The ArticleSiteMapData provider looks like this
1: /// <summary>
2: /// A SiteMap provider for the Article page.
3: /// </summary>
4: public class ArticleSiteMapData: DynamicDataSearchSiteMapProvider
5: {
6: public ArticleSiteMapData() {}
7:
8: public override IEnumerable DataQuery()
9: {
10: // Holds a list of the ArticleEntry's
11: List<SiteEntry> list = new List<SiteEntry>();
12: TList<Article> articles = new TList<Article>();
13:
14: articles = DataRepository.ArticleProvider.GetAll();
15:
16: foreach (Article article in articles)
17: {
18: string url = BLL.URLRewrite.GetArticleRewrite(article);
19: SiteEntry siteEntry = new SiteEntry(url, article.AddedOn, SiteEntry.ChangeFrequency.Daily, "0.5");
20: list.Add(siteEntry);
21: }
22:
23: articles = null;
24: return list.ToArray();
25: }
26: }
Lines 14 - 21 you would replace with your code get your dynamic data. Lines 19 and 20 is where the SiteEntry object is created and added to the List.
So how does the List<SiteEntry> tie into the provider configuration? The answer is in the attributes/properties that are defined for provider. For this sample we have targetUrl ="", targetUrlSeparator="/", queryStringDataFormatString="{0}" and queryStringDataFields="Url" this tells the provide to build a string with the base of targetUrl, then add targetUrlSeparator, followed by calling string.format using queryStringDataFormatString with the parameters equal to the value of the properties listed in queryStringDataFields. In other words the url will look like this.
http://www.yourdomain.com/<value of url field>
In my site, I generate the Url in code so that it will be consistent between different pages. A "real" world example might look like this.
Properties: targetUrl ="BookReviews", targetUrlSeparator="/", queryStringDataFormatString="{0}/{1}.aspx" and queryStringDataFields="Id, Title" which would generate a url similar to this.
http://www.yourdomain.com/1/title.aspx
Notice that I had two field in the queryStringDataFields property. You can list as many as you want, they just need to be comma separated. This means that the SiteEntry object needs to have at least the two properties of Id and Title.
For more information
That's about it. I was a lot to read and digest but hopefully useful. I you have any questions, please feel free to contact me or leave a comment or you can discuss the Sitemap provider in the ASP.net forums.
Search Engines were you can submit a feed
Here some search engine urls where you can submit you site maps.
Google: https://www.google.com/webmasters/tools/siteoverview
Yahoo: https://siteexplorer.search.yahoo.com