Monday, December 26, 2011

SharePoint 2010 Search versus KnowledgeLake Search (Press the Easy Button) Part Two

This is part two of a three part series comparing SharePoint Search with KnowledgeLake Search.  KnowledgeLake Search has many features you just won’t find in SharePoint Search and features that are comparable to SharePoint out of the box but are not easy to use. You could try implementing some of these features on your own, but KnowledgeLake Search has years of experience enhancing SharePoint Search.

KnowledgeLake Search to SharePoint Search Features


KnowledgeLake Search

SharePoint Search

Metadata searching



Saved searches



Multiple searches



Filtering results



Sorting Yes Yes
Grouping Yes No

Search Center



Web Parts



Custom columnar results Yes No
Real time user interface customization Yes No

Localization of managed properties



In-line editing of search results



Document preview



Advanced options






Exporting results Yes No
Email links and attachments Yes No
Federated search results No Yes
Query suggestions No Yes

In this post I will cover the features highlighted in red. In part one I covered Metadata searching through Grouping.

KnowledgeLake Search vs SharePoint Search Part One

Search Center

Both SharePoint and KnowledgeLake offer a Search Center site template. Many people ask what exactly is the benefit of a Search Center.

Following search UI best practices, the Search Center provides a consistent UI for different types of searches. The SharePoint Search Center comes with a tab control that enables you to organize links to other search pages. Each tab can represent a different type of search for users. For example, some types of searches might only need a standard search box and search results. Other types of searches might require a more advanced search that includes property searching along with a refinement panel with the results. Each tab enables you to customize the search and the results UI using the SharePoint Search web parts. Customization is easy thanks to customizable search pages that are based on a search template.

KnowledgeLake also provides a Search Center site template. The template provides a consistent UI for searching. However, customization is not needed because all features for effective searching are available to the user and an administrator is not needed. Term and property searches, scoping of searches including path scoping and the ability to select what data to display in the results are all customizable by the user. The KnowledgeLake Search Center lets users have as many ad hoc and saved searches open as they want. Users can see the search criteria and the results on the same page. The search center allows users to save, share, and export searches.



Web Parts

SharePoint offers a plethora of search web parts. These include a standard search box, advanced search, standard search results, search paging and refinement panel. The many web parts gives an administrator a great amount of flexibility in designing custom search pages to fit user’s needs. A great place to learn more about all the different SharePoint search web parts is in the “SharePoint 2010 Enterprise Content Management Implementers’ Course” on TechNet. Check out module 8 “Search User Interface”.

KnowledgeLake Search includes two web parts. One is the Query Builder web part. This web part displays a saved query that was built from the KnowledgeLake Search Center.  As a search manager you can build a custom search with terms, metadata properties, scopes, and advanced options. The web part’s configuration can further customize the search by allowing a designer the  ability to hide or disable criteria being used in the search. Compared to the SharePoint Search web parts, KnowledgeLake’s Query Builder web part is very easy to build a custom search without having to connect and configure multiple web parts. SharePoint Search web parts requires users to have knowledge of XML and XSLT to customize functionality.

With KnowledgeLake Search In order to see the results you must deploy the “Query Results” web part. This is the same results control you see in the KnowledgeLake Search Center with all the rich UI functionality, including thumbnails and the ability to edit properties in-line. There is no configuration needed if you locate it on the same web part page as the Query Builder web part. This is a big advantage over SharePoint Search because deploying the SharePoint Core Search Results web part on the same page as the Advanced Search web part is virtually impossible. The Query Results web part also supports a URL based API. This enables the Query Builder web part to redirect a search to the Query Results web part using a URL similar to the SharePoint Core Search Results web part. The KnowledgeLake Search web parts can also be used to customize SharePoint Search Centers.


Custom Columnar Results

Having the ability to customize what data is returned in search results is a must for any ECM search solution. The data returned in a search results is the only information a user has to determine if the result is what they are looking for. Site administrators can customize the SharePoint Core Search Results web part to add additional properties to be returned in the results. Unfortunately, site administrators must have a working knowledge of XML and XSLT. Below is a graphic of the configuration pane for the SharePoint Core Search Results web part.

Customizing which document properties are displayed in search results and how they are displayed is accomplished by modifying an XML and an XSL template. The first step is to uncheck the Use Location Visualization check box. This disables the use of the default XML for viewable document properties and the XSL template for rendering results. This makes it easy to revert back to SharePoint’s default display of search results.

The second step is to add any document properties you want displayed to the Fetched Properties text box. This text box contains XML containing column elements that define the names of document properties returned with search results. The final step is to edit the XSL template to display your document properties and manipulate how they are displayed. Clicking on the XSL Editor button displays a Web dialog containing the XSL template. This can be copied to a text editor where you can modify it. After modification, copy the edited text back into the dialog and save it. As you can see this is not an easy process. Also the document properties are not displayed in a columnar order making it difficult for users see and use in comparison with other results.



The KnowledgeLake Search Center easily allows for adding whatever available document properties to your search results. Users (not Site Administrators) just need to click on the button in the result columns pane and then drag and drop document properties to be displayed. This dialog also lets users position the properties in order that they want them displayed. The properties are displayed in columns making it easy to see and compare. The customized results can be saved and then  used from any KnowledgeLake Search Center or web part.



Real Time User Interface Customization

One of the most important aspects of an effective ECM search solution is the ability to manipulate search results to make is easy for users to find what they are looking for. SharePoint Search has the Refinement Panel web part which allows users to select certain categories of document properties and the search results will be filtered to that category. This web part is also difficult to configure by a site administrator and requires knowledge of an xml schema. The user has no ability to manipulate the search results.

KnowledgeLake Search does allow for users to manipulate search results. The user can drag column headers to the top of the search results to group by one document property or multiple. The grouping also show the counts of results in the group. Users can also re-position and re-size columns to make it easier to compare results. Sometimes when grouping by dates users may not want to include the time portion of the date or the decimal precision of a number is too long. Users can edit the result columns format. They can choose not to display the time of a date property or limit the number of decimal places.


Localization of Managed Properties

Managed properties are the document properties used when searching for documents in an ECM solution. SharePoint provides the ability for search administrators to create managed properties using names that make sense to the business and users. For example, if a business has multiple departments that have document properties called “Customer Number” and “Customer ID”, then a search administrator can map both these document properties to a managed property called “CustomerNumber”. This provides a consistent property search across departments. However one of the biggest complaints about the naming of managed properties is the SharePoint Search restriction of not being able to put spaces in the names. In the example given it would be easier for users to remember the name by using “Customer Number” instead of “CustomerNumber’”. KnowledgeLake gives administrators the ability to name managed properties any way they want. Administrators can define a resource file to map a managed property name to any display name desired. So if you have users who want to search using “Invoice Number” instead of “InvoiceNumber” then this is not a problem. Another problem is localization of managed properties. So if you have a French SharePoint site then they will be forced to use the built in English managed properties. Searching can become confusing for mixed language farms. Using the resource file mapping provided by KnowledgeLake Search, administrators can define language based resource files that map managed properties to a localized name. The KnowledgeLake Search center and web parts will display the localized managed property helping your international users find what they are looking for.

In-Line Editing of Search Results

An important search best practice is giving users the ability to take action on the search results. For instance, the user has already spent time looking for the document so a good ECM search solution should give the user the ability to view, check in, check out, declare as a record or even edit the properties of the document. Of course KnowledgeLake Search gives you a context menu for each result to execute such actions without having to navigate away from the search results. This makes for a very efficient use of search. Below is a picture of how you can expand a search result to display a property edit form. The form can take advantage of any KnowledgeLake Index behaviors set up to enhance indexing of the document.


Document Preview

One of the most useful search feature is the ability to preview a document within the search results. This helps users find documents by providing a glimpse into the document. SharePoint Search does not provide this, however, SharePoint Fast Search does provides a thumbnail of the first page  for Microsoft Word and PowerPoint documents. KnowledgeLake Search provides a “Thumbnail Viewer” in the search results enabling users to page through and visually inspect the complete document within the results. The best part about this viewer is the ability to view many document formats such as Microsoft Word, PowerPoint, Excel, MSG, PDF, TIF, JPG, BMP and PNG. This feature is a powerful enhancement for any ECM search solution.


Why would you want to search SharePoint with anything else?

This concludes the second part of a three part series of comparing KnowledgeLake Search to SharePoint Search. The features discussed in this post shows KnowledgeLake Search is much more efficient at helping users find what they are looking for. Your users are not searching SharePoint just to search but to accomplish a task. An effective search makes your organization productive letting your users spend less time searching for information. For example, It is estimated an effective search strategy can save approximately 3.7 hours per day per worker finding information*. This can translate to hundreds of thousands of dollars annually depending on the number of information workers and the hourly cost. The ability to let users dynamically add metadata, group, sort and fully preview documents when searching will increase search productivity substantially over the out of the box SharePoint Search. Being able to accomplish this without a site administrator and knowledge of XML and XSLT is a great advantage. Finally, allowing users to take action on search results directly speeds up the time to accomplish a task. Searching in SharePoint should be easy. It is is with KnowledgeLake Search.

*Source: IDC’s Information Worker Productivity Survey, October and December 2008 LinkedIn Survey

Monday, October 31, 2011

SharePoint 2010 Search versus KnowledgeLake Search (Press the Easy Button) Part One

Technorati Tags: ,,,


I have been working on KnowledgeLake Imaging 4.2 for the last seven months. One of the main components of this product is our Search. Search is one of the most important parts of an effective ECM solution. KnowledgeLake Search adds incredible value to the out the box capabilities of SharePoint. In fact I would not want to use SharePoint with out it. KnowledgeLake Imaging is not just about scanned documents. The features we have provided in this product can be used with all content in SharePoint regardless of the source. Many have said that the features we implement can be done with out the box SharePoint. I agree, because we have done it. However, it will take an incredible amount of time and money to accomplish this. KnowledgeLake Search has too many features to list in just one posting. This is part one of comparing KnowledgeLake Search to SharePoint Search (MSS) showing you how KnowledgeLake Search makes SharePoint searching much easier.


KnowledgeLake Search to SharePoint Search Features


KnowledgeLake Search

SharePoint Search

Metadata searching



Saved searches



Multiple searches



Filtering results



Sorting Yes Yes
Grouping Yes No

Search Center



Web Parts



Localization of managed properties



In-line editing of search results



Document preview



Advanced options






Exporting results Yes No
Email links and attachments Yes No
Custom columnar results Yes No
Real time user interface customization Yes No
Federated search results No Yes
Query suggestions No Yes


Metadata Searching

Searching with properties is not easy in SharePoint. First you must know the name of the managed property. Second there is no way to communicate to users the catalog of managed properties available to them. Users must guess the name of the managed property and they must understand the syntax of searching. For example using the standard search box in SharePoint a user would need to type in the name of the managed property along with a colon and then the term.

SharePoint search becomes even more difficult when users try to execute complex searches such as date ranges. For example, the following keyword search retrieves documents created between January 01, 2011 and February 01, 2011.

created>01/01/11 AND created<02/01/11

This seems somewhat easy. However, if the user put any spaces between created and the greater than sign, then the search does not work. The same applies if the user puts a space between the greater than sign and the value. Furthermore, if the user were to use a lower case AND conjunction the search fails. More uncertainty whether the search will work arises when introducing multi-culture date formatting. Out of the box SharePoint search requires training.

Metadata searching becomes a little easier if your users take advantage of the Advanced Search Web Part. With SharePoint advanced search you can make it easier by using the property restriction section. Here users can chain together search criteria including ranges.

The biggest problem with the Advanced Search in SharePoint is that it requires a site administrator to configure the web part for the managed properties to show up. This involves getting trained on how the web part works and the formatting and use of XML.

KnowledgeLake metadata searching is much easier. All managed properties are available to users to choose from without the users having to memorize nor an administrator having to configure. The user can just select the managed property from a drop down list and easily enter in a date range search using a range operator and a date picker.


Saving Searches

You can save a search in SharePoint but it is more like consuming a search rather than saving it. SharePoint offers a way to receive an alert if items in a search change. The alert can be sent either as an email or text message. You can navigate back to search results from the email. You can also consume a saved search as an RSS feed. Unfortunately, neither of these ways of saving searches describe what you searched on. You can put a description around it but it is not easy to review the search criteria. If you built a somewhat complicated search using Advanced Search, is no way back to Advanced Search in order to tweak the search and re-save it.

KnowlegeLake Search allows you to save, share and re-use searches. In contrast to SharePoint, KnowledgeLake’s Search Center allows user’s to load saved searches. You can also assign security around the sharing of searches. Complex searches can be built once and re-used among groups in your enterprise. This can speed up the search process for your users.


Multiple Searches

SharePoint does not support having multiple searches open on one page. However, you can have multiple searches in separate tabs in the browser. This can become tedious where the user must navigate multiple times to a SharePoint Search Center and execute a search. KnowledgeLake Search is much easier allowing the user to open unlimited number of saved searches in multiple tabs in the search center. The user can switch between each tab changing the criteria and seeing the results.


Filtering Results

SharePoint enables the filtering of results using the Refinement Panel web part. The web part will display categories of managed properties along with a certain number of values the user can click which will filter the results. Unfortunately, this web part is difficult for a site administrator to configure. Once again the site administrator must know XML and be trained how to modify it  in order for the managed property categories to show up. You can set up hard coded ranges for dates, but the XML web page dialog is difficult to edit. Of course you could copy and paste this XML into some XML editor and paste it back. The point is that this is not easy.

KnowledgeLake lets user easily filter their results with out any site administrator configuration. The user has unlimited abilities to filter their result with all the values available and with any operator. Users can even chain the filters together.



SharePoint Search can only sort results by the last modified date or relevance. By default the results are returned sorted by the most relevant. Relevance is a calculated value during query execution and is based on a complicated model that I have posted about previously. Many times enterprises need different types of sorting without having to change relevance models. The process of building and implementing custom ranking models is difficult. KnowledgeLake Search helps users find what they are looking for quicker by allowing ad hoc sorting. Users can easily click on a results column to sort ascending, click again to sort descending. Users can even do multiple column sorting by holding down the shift key and clicking on another column. What is even better is users can save this custom sorting with the query to be re-used.


SharePoint Search supports grouping through the Refinement Panel web part mentioned previously, but this web part is difficult to configure and does not allow users to dynamically group results. KnowledgeLake Search enables users to drag multiple columns to the results header to group results. These groupings also show counts of the items within and can be sorted. Once again this adds to the user’s search productivity by letting users dynamically manipulate their results. Another great feature is that the grouping can be saved with the search to re-used.


This is the end of Part One of the comparison of SharePoint Search and KnowledgeLake Search. I hope you can see just in this brief overview the value KnowledgeLake Search adds to SharePoint Search.  Trying to add these features to out of the box SharePoint can take a substantial amount of time and money, let alone, the time to test and debug such a solution. In Part Two and possibly a Part Three I will cover the rest of the features.

Tuesday, September 27, 2011

SharePoint 2010 Search Query Suggestions Explained

Technorati Tags: ,,

A principle of interface design is to provide the user with feedback about the status of the system and how that relates to the user's interactions with the system. Searching can be mentally intensive and feedback about query formulation, such as the reasons the particular results were retrieved, and what next steps can be taken can help the user easily find what they are searching for. One of the ways SharePoint Search gives feedback is the feature called “Query Suggestions”.  You are probably familiar with this feature if you have ever searched using Google or Bing.


SharePoint query suggestions come in two forms, pre-query and post-query.

  • A pre-query suggestion is type-ahead feature that appears under the search box as shown above.

  • A post-query suggestion is similar except that it appears after the search has been submitted. It appears as a “Did you mean” link; by clicking it, the query is then executed and the results displayed.


Both types of query suggestions adhere to the search UI best practice of helping users search efficiently.


What determines a query suggestion?

Query suggestions are based on frequently used search terms and click through to results. Listed below are the steps needed for SharePoint to promote a term to a query suggestion.

  1. The associated Search Service application must have query logging enabled.
  2. The user must type in a valid word or set of words in the search box.
  3. The user must click on 6 or more search result links and open or save the file. Some types of pages containing the term can qualify as click-throughs.
  4. The Query Logging timer job must be run.
  5. The Prepare Query Suggestions time job must be run.

You can also through Microsoft Powershell promote query suggestions which will make them appear immediately.

#Execute in SP2010 PowerShell console

#Get the name of the Search Service Application

$searchapp = get-SPEnterpriseSearchServiceApplication

#Add your query suggestion


SearchApplication $searchapp –Language EN-US –Type

QuerySuggestionAlwaysSuggest –Name “Document”

#Run the timer job

Start-SPTimerJob –Identity “Prepare Query Suggestions”


Enabling query suggestions

To enable pre-query suggestions edit the “Search Box” web part, and in the “Query Suggestions” section check the box “Show Query Suggestions”.  You can also adjust the delay in milliseconds before the drop down list appears and you can limit the number of suggestions shown.


To enable post-query suggestions navigate to your results page and edit the “Search Box” following the same instructions as before.

Adding query suggestions to your own search applications

The “Search Box” web part makes an ajax call to the Search.asmx web service’s GetQuerySuggestions method passing in the term on each keystroke. Your own search applications can do the same using jquery. You may want to do this from the server side object model. This can be accomplished using the KeywordQuery class in Microsoft.Office.Server.Search. Below is an example.

public static StringCollection GetQuerySuggestions(string term)

    StringCollection suggestions = null;

    SPServiceContext context =

    SearchServiceApplicationProxy searchProxy = context.
        as SearchServiceApplicationProxy;

    using (KeywordQuery query = new KeywordQuery(searchProxy))
        query.QueryText = term;
        suggestions = query.GetQuerySuggestions(5, true, false, true);

    return suggestions;

The GetQuerySuggestions method takes four arguments. The first is the the maximum number of suggestions you want to return. The second is whether the suggestions are for the term the user is typing in the search box or for a completed query. If you set this to false you must set the KeywordQuery.QueryProperties.QueryText. This tells the code that you want post-query suggestions. The third argument tells the code that you want html encoded strings bolding the parts of the term that the user typed in. The last argument sets whether the suggestions should be capitalized.

Suggesting a best practice

SharePoint Search provides Search UI components and a rich set of Search Web Parts that enable many configuration options. Many of these options relate to Search UI best practices. Query suggestions provide informative and efficient feedback. Understanding the best practices helps you make the correct decisions when customizing the Search UI and ultimately making your ECM implementation much more “findable”.

Wednesday, August 31, 2011

SharePoint 2010 Code Tips – Setting a Managed Metadata Field with the Client Object Model

Technorati Tags: ,,

I have seen many requests on how to set a managed metadata field using the client object model. Most of the questions revolve around the fact that many remote applications do not have access to the required information for setting a managed metadata field. A managed metadata field contains a term and the term is stored with three pieces of information. The first piece is the ID that represents the ID of the term in the local site collection’s TaxonomyHiddentList. The second piece is the term value itself which is usually the label the user tagged it with. The final piece is the term’s ID which is in the form of a GUID. This represents the unique ID of the term in the term store of the site collection. This term store is published from the associated metadata service application. Here is an example of how the value is stored for a term named escalade.


Remote applications have the term they want to add but do not have the ID to the TaxonomyHiddenList  list and the unique ID of the term. The ID to the term in the TaxonomyHiddenLIst can be ignored by just using –1 in its place. However, you must have the ID of the term. One solution was to query the TaxonomyHiddenList list for it. Unfortunately, the term is only put there after it has been used in the site collection. This is done for performance reasons.  In order to set the managed metadata field from CSOM you must use the TaxonomyClientService web service to obtain the ID of the term. In this post I will show you how to use both the managed CSOM and the TaxonomyClientService web service to set the managed metadata field, also, I will show you how to do this with a multi-valued managed metadata field.

Getting all the information

Setting a managed metadata field requires multiple steps in order to gather the required information.

  1. Get the managed metadata field of the list.
  2. Get the shared service ID (GUID of the metadata service application) and the term set ID from the field’s schema.
  3. Get the set of terms for the term set from the TaxonomyClientService web service.
  4. Parse the returned xml and obtain the term ID for the term you are going to use.

Four simple steps. I wish they were. The biggest problem is understanding how to parse the returned xml from the GetTermSets method of the TaxonomyClientService. The xml schema has no discernable reason or order to it. The schema has arbitrary attribute names. Below is an example:


The term set I am using in this example is called Cadillac and contains a group of terms for different models of Cadillac cars. Each term is contained in a “T “ node and the “a9” attribute represents the ID of the term. The term label is contained in the child “TL” node and the label is denoted by the “a32” attribute. The code below calls the TaxonomyClientService and  searches for term using the term label sent in as an argument.

public static string GetTermInfo(string siteUrl, Field field, string term, ref string textField)

string sspId = string.Empty;
string termSetId = string.Empty;

XElement schemaRoot = XElement.Parse(field.SchemaXml);

foreach (XElement property in schemaRoot.Descendants("Property"))
XElement name = property.Element("Name");
XElement value = property.Element("Value");

if (name != null && value != null)
switch (name.Value)
case "SspId":
sspId = value.Value;
case "TermSetId":
termSetId = value.Value;

case "TextField":
textField = value.Value;


string termSetXml = GetTerms(siteUrl, sspId, termSetId);
XElement termSetElement = XElement.Parse(termSetXml);

var termId = from t in termSetElement.Descendants("T")
where t.Descendants("TL").Attributes("a32").First().Value.ToUpper() == term.ToUpper()
select t.Attribute("a9");

if (termId != null && termId.Count() > 0)
return termId.First().Value.ToString();
return string.Empty;


private static string GetTerms(string siteUrl, string sspId, string termSetId)

termsservice.Taxonomywebservice ts = new termsservice.Taxonomywebservice();
ts.UseDefaultCredentials = true;
ts.Url = siteUrl +"/_vti_bin/taxonomyclientservice.asmx";

string timeStamp;

string termSetXml = ts.GetTermSets(WrapXml(sspId.ToString()),
WrapXml("0"), out timeStamp);

return termSetXml;


private static string WrapXml(string value)
return string.Format("<is><i>{0}</i></is>", value);


Setting all the information

The code below puts it all together by setting the required information. In the case of using the client object model you must set two fields for the update to be successful. Every managed metadata field has a corresponding hidden taxonomy text field. The GetTermInfo method returns the ID of this hidden field so the code can also update its value. This field holds the same information as the managed metadata field but is used for internal purposes. Notice also the code checks to see if the current value supports multiple values by checking if the type can be assigned to an object array. If it can then you must add terms to a  list an convert the values back  to an array. You must also append all the values with a semi-colon for the hidden taxonomy field. Finally you will only need to set the ID value to a –1 and the server side code will look it up for you. So there are no extra calls needed to look up the value from the TaxonomyHiddentList.

public static void SetManagedMetaDataField_ClientOM(string siteUrl,string listName, string itemID, string fieldName, string term)

ClientContext clientContext = new ClientContext(siteUrl);
List list = clientContext.Web.Lists.GetByTitle(listName);
FieldCollection fields = list.Fields;
Field field = fields.GetByInternalNameOrTitle(fieldName);

CamlQuery camlQueryForItem = new CamlQuery();
string queryXml = @"<View>
<FieldRef Name='ID'/>
<Value Type='Counter'>!@itemid</Value>

camlQueryForItem.ViewXml = queryXml.Replace("!@itemid", itemID);

ListItemCollection listItems = list.GetItems(camlQueryForItem);


string hiddentTextFieldID = string.Empty;
string termId = GetTermInfo(siteUrl, field, term, ref hiddentTextFieldID);

if (!string.IsNullOrEmpty(termId))
ListItem item = listItems[0];
string termValue = string.Empty;
string termHTVal = string.Empty;
object itemValue = null;

List<object> objectVals = null;

if (item[fieldName] != null && item[fieldName].GetType().IsAssignableFrom(typeof(object[])))

termValue = term + "|" + termId;
objectVals = ((object[])item[fieldName]).ToList<object>();
itemValue = objectVals.ToArray<object>();

foreach (object val in objectVals)
{ termHTVal += "-1;#" + val + ";"; }
termHTVal = termHTVal.Substring(0, termHTVal.Length - 1);

termValue = "-1" + ";#" + term + "|" + termId;
termHTVal = termValue;
itemValue = termValue;


Field hiddenTextField = fields.GetById(new Guid(hiddentTextFieldID));

item[hiddenTextField.InternalName] = termHTVal;
item[fieldName] = itemValue;




Advanced Scenarios

In summary this example will take the term label, site URL, managed metadata field name, list name and the ID of the item you want to update. It will use this information to update the managed metadata field with the term. You can use this example to build more sophisticated scenarios where you may want to replace a particular term with another if the column holds multiple values. The taxonomy web service has methods to lookup terms based on a culture for multi-lingual cases, and the ability to search for terms where there are more than one term store. Your best bet is to build a term picker using the information from the taxonomy web service. In this case users would pick from a dialog what term they want to tag a value with. Using a term picker the term ID would already be available to use and would allow the user to easily navigate complicated term stores.

I hope this helps in understanding all the extra steps that must be taken when setting managed metadata from remote applications. Hopefully, this will become much easier in future versions of SharePoint.

Thursday, August 18, 2011

SharePoint 2010 Code Tips – Client Object Model -Add a Site Column to a List

Technorati Tags: ,,

I am going to start sharing more of the code I post on the MSDN forums. These postings will always begin with SharePoint 2010 Code Tips. I will do my best to explain what scenarios the tip may be useful for. With the advent of Office 365 there is a rising interest in accomplishing standard tasks remotely. For instance, adding a site column to an existing list. If you are developing a remote application for doing simple site administration this can be a common task. Many developers know how to accomplish this using the server object model, but many are not familiar on how to do it remotely. In this posting I will show how to accomplish this using the managed Client Object Model and also using both the Webs and Lists out of the box web services.

Add a Site Column with CSOM

The code below uses the managed CSOM to locate a site column and add it to an existing document library. It finds the site column by filtering the Site’s AvailableFields FieldCollection using a lambda query. The second step is to get the document library you want to add the column to by using the Web’s ListCollection GetListByTitle method. Next add the Field to the List’s Fields collection. Finally, update the List and call the ClientContext’s  ExecuteQuery method.

public static void AddSiteColumn(string siteColumnName)
    ClientContext context = new ClientContext("http://basesmc2008");
    Web site = context.Web;
    FieldCollection collFields = site.AvailableFields;       

    var siteColumn = context.LoadQuery(collFields.Where
        (c => c.Title == siteColumnName));


    if (siteColumn != null && siteColumn.Count() == 1)
        List list = context.Web.Lists.GetByTitle("tester2");


Add a Site Column with Web Services

Those of us who have been developing on SharePoint since 2003 know how important using SharePoint web service can be. Even though the CSOM has made the coding simpler many tasks can still be done using out the box web services. The code sample below starts by using the Webs GetColumns method to return a list of site columns. Using xml linq you obtain the complete field node. Next add the field node to the list using the Lists UpdateList method. A required step to update the list is to get the current version of the list. The current version must be sent in the UpdateList  method.

using System;
using System.Collections.Generic;
using System.Xml;
using System.Xml.Linq;

public static XmlNode UpdateListAddSiteColumn(string siteColumnName)

    XmlNode resultNode = null;

    webs.Webs webs = new webs.Webs();
    webs.Url = "http://basesmc2008/_vti_bin/webs.asmx";
    webs.UseDefaultCredentials = true;

    XmlNode columnsNode = webs.GetColumns();

    XElement xColumns = XElement.Parse(columnsNode.OuterXml);

    var siteColumn = from t in xColumns.Elements()
                     where t.Attribute("DisplayName").Value == siteColumnName
                                                    select t;

    if (siteColumn != null && siteColumn.Count() == 1)
        string xml = "<Batch OnError='Continue'>";
        xml += "<Fields><Method ID='1' Cmd='New'>";
        xml += "!@siteColumnSchema";
        xml += "</Method></Fields></Batch>";

        xml = xml.Replace("!@siteColumnSchema", siteColumn.First().ToString());

        listservice.Lists listProxy = new listservice.Lists();

        listProxy.Url = "http://basesmc2008/_vti_bin/lists.asmx";
        listProxy.UseDefaultCredentials = true;

        XmlNode listNode = listProxy.GetList("tester2");
        XmlNode version = listNode.Attributes["Version"];

        XmlDocument doc = new XmlDocument();

        XmlNode newFieldsNode = doc.SelectSingleNode("//Fields");

        resultNode = listProxy.UpdateList("tester2", null, newFieldsNode, null, null, version.Value);

    return resultNode;



As you can see the CSOM example is much easier than the web services example. Keep in mind that the adoption of SharePoint is now becoming so widely adopted, that many non-.Net developers are trying to integrate with it. Many of these developers are familiar with using web services, so the death of Web 2.0 web services in SharePoint is still a long way off.

Tuesday, August 16, 2011

The Microsoft SharePoint 2010 Enterprise Content Management Implementers Course

Technorati Tags: ,

Microsoft TechNet is a great place to get easy to use, timely and free information to implement new technologies. For example,  Silverlight, SharePoint 2010 Development and now how to implement SharePoint 2010 as an ECM platform. The new SharePoint ECM Implementers Course teaches implementers how to use SharePoint Server to implement an Enterprise Content Management (ECM) system. ECM systems are designed to manage large amounts of content. This content can include documents , wiki libraries, blog posts, and other types of non-document content.  This course develops the key skills that are necessary to deploy SharePoint Server for ECM solutions at organizations of any size.

This course consists of 15 in depth modules including, Infrastructure and Storage, Information Architecture, Content Types, Metadata Taxonomy, Implementing and Managing Search, and Records Management. The course was put together by Paul Andrew and Rob Bogue. The modules were produced by SharePoint MVPs Eric Shupps, Darrin Bishop, Ben Robb, Rob Bogue and myself. The modules were constructed with great care to make difficult subjects easy to understand.  The goal of the course is to help you clearly understand how to leverage SharePoint features to implement an ECM system.  I can attest to this because there was substantial re-work done before it could be considered easy to understand.

I learned a great deal from the process and Rob Bogue who taught me how to improve on taking a difficult subject and making it easy to understand.

I recommend you take the time to download the course and watch  it. The hands-on labs are straight forward and understandable. The course will give you a clear path on how to use SharePoint as an ECM platform.

SharePoint ECM Implementers Course

Saturday, July 23, 2011

SharePoint 2010 Search Scopes Explained

Technorati Tags: ,,

Search scopes enable users to narrow their searches based on content sources, web addresses, and metadata. This makes it easier for the user to search. Scopes are an integral part of the search user interface allowing users to choose a scope before performing a search query. Scopes can also be used to pre-configure search web parts such as the Search Results Web Part.

What is a Search Scope?

Scopes can be considered pre-built search criteria that are logically “AND” to users’ searches. They represent specific content or topics that are common or important to users. For example, you can define a scope for items that relates to a particular group in the organization such as Human Resources (HR) or Marketing. You can also create a search scope that encompasses several other scopes. In addition, you can set search scopes at both the search service application level and at the site administration level. Search scopes set at the service application level are available to all sites and site collections within the service application. These scopes cannot be modified or deleted at the site administration level.

A scope is made up of a title that the user sees when choosing it from a Search Web Part. The description describes what the scope represents. This is for administrative purposes. The Target Results page gives you the ability to use the Default Results page or specify a Custom Results page.

The rule represents the condition or criteria for defining the subset of content. Rules can have different types of conditions and behaviors. An administrator can create a complicated scope by chaining together different rules. This lets users easily append complicated query conditions to their searches.

Scope Rule Types

There are four search scope rule types.

  • Web Address Rules -  This type of rule can incllude content in web sites, file shares, exchange public folders, or any other content in the search index that has a URL. These can include the following.
    1. Folder - Includes items in the folder and subfolders that pertain to the indicated path.  With SharePoint URLs you can create rules that include or exclude different elements, such  as a site or a particular folder.
    2. Host Name - Limits content to a particular server name.
    3. Domain or Sub Domain - Limits content to a specific, fully qualified domain name such as
  • Property Query Rules - Limits content where a managed property is equal to a certain value. Only managed properties that have the “Allow this property to be used in scopes.” option checked can be used to limit content.
  • Content Source Rules - Enables you to limit content to a particular content source. This is useful if your service application has numerous content sources and you want to limit the scope to just a SharePoint content source or more granular types of content sources such as My Sites or Team Sites.
  • All Content Rules - Includes all content. Enables the construction of a more complicated scope that includes many rules. For example, you can combine this rule with an Exclude Content source rule. By doing so, the scope contains all content except possibly a file share content source.

Scope Rule Behaviors

Scope rule behaviors enable administrators to combine multiple rules together using simple logic. This simple logic enables administrators to create precise subsets of content. The recommended limit for the number of rules per scope is 100 or 600 per service application.

There are three types of rule behaviors:

  • Include - Any item that matches this rule is included, unless the item is excluded by another rule. Use this option to apply an “OR” rule.
  • Required - Every item in the scope must match this rule. Use this option to apply an “AND” rule.
  • Exclude - Items matching this rule are excluded from the scope. Use this option to apply an “AND NOT” rule.


Search Scope Example

An administrator can create multiple rules for a scope to define a precise set of content.

The above example illustrates how SharePoint evaluates the rules contained in a scope and determines what content is included.

All Include rules are applied with an OR; the required rules are applied with an AND; and the Exclude rules are applied with an AND (NOT ..).

For example, as a site administrator you can create a scope to limit a user’s query to match only documents in the team site, or have a title of Monthly Budget, and are not located in the archive site.

Rule Type



Web Address




Title = “Monthly Budget”



IsDocument = 1


Web Address



If the user searches for a keyword of April along with the above scope, the actual query would look like this:

April AND ((Title = ‘Monthly Budget’ OR CONTAINS(Path, ‘http://servername/teamsite’) AND IsDocument = 1 AND NOT CONTAINS(Path, ‘http://servername/archivesite’))

SharePoint search scopes effectively build complicated filter conditions to determine the subset of content. They are logically appended with an AND operator to the user’s query. The user does not have to remember the complicated conditions to append the query, but only needs to remember the name of the scope.

Types of Scopes

There are two types of Scopes in SharePoint

  • Shared - Defined in the Search Service Application and can be used by all site collections in the farm associated with the Search Service Application.
  • Site level -  Defined at the site collection level by a site collection administrator and can be used only in the site collection

Shared scopes can be used across the farm by all site collections by defining the scope using the Search Service Application in Central Administration. The shared scopes are propagated to the site collections through a scheduled timer job, after which they become available to other site collections.

You can control what shared scopes are displayed in the site collection from the Site Collection Administration Search Scopes page.

On this page you can control what scopes can be displayed in either the Search Drop Down or Advanced Search Web Parts. This page can also be used to define site collection scopes that can be used only within the current site collection and sub sites. Shared scopes cannot be modified or deleted at the site administration level.

How Scopes Affect Search

Using scopes in a query can affect the performance. The default behavior is to execute scope rules during query execution when the number of rules is less than 25. This can increase the time it takes to return results. Scopes with rules greater than 24 are compiled and cached.

To speed up query performance, use Windows PowerShell to set a scope’s CompilationType to AlwaysCompile. However, this affects crawl performance because rules are evaluated during the crawl.

Windows PowerShell example:

$app = Get-SPEnterpriseSearchServiceApplication
$scope = $app | Get-SPEnterpriseSearchQueryScope -Identity "MyScope“
$scope.CompilationType = “AlwaysCompile“


How Scopes Affect User Search

Scopes enhance the user search process by making it easier for users to construct searches. Users can append keywords to a selected scope.


Complicated scopes can be constructed that make it easy for users to append keywords or property queries to a selected scope. Users do not need to remember all the rules behind a scope. Instead they need only recall the name of the scope and what it represents. This helps users find what they are looking for by segregating content into manageable and understandable subsets.

In addition, scopes can be used to direct users to custom search results pages to further enhance the search experience. However, to avoid confusion, care must be taken not to create too many scopes. Users must be educated on what each scope represents in order to use scopes effectively. Limit the number of scopes in the drop-down list to no more than 10.


This post was about explaining the makeup of a SharePoint Search Scope and benefits of using them in your search user interface. This is limited to SharePoint Search and does not cover FAST search.  There is a lot of content about how to manage and configure scopes but nothing about the purpose of using them. Hopefully, this help illuminate the benefits and encourage you to use them.

Tuesday, July 19, 2011

SharePoint Server MVP 2011

Technorati Tags: ,,

Earlier this month I was notified that I had been awarded Microsoft’s MVP for SharePoint Server for 2011. This is my third year in a row. I am proud to be part of this awesome community. Being able to share my knowledge in the MSDN forums is a rewarding experience. The forum is a great place to help others get the most out of an amazing Microsoft product. SharePoint 2010 is now being recognized as a legitimate ECM platform. Also, I am glad I am part of a great team of developers at KnowledgeLake. KnowledgeLake is always looking to add value to SharePoint as an ECM platform, and for that I am thankful.

Friday, June 24, 2011

Advanced Search for SharePoint 2010 Hold and eDiscovery

Technorati Tags: ,,,,

In the last ten years “eDiiscovery” has become more important with many companies dealing with litigation. Courts can require companies to search and discover evidence within electronic documents. It is the responsibility of the company to put these documents on “hold”. “Holding” a document can lock the document in place, preventing it from being edited, moved, deleted, or checked out. The document can also be copied to an authorized document or record center.  Hold and eDiscovery is a very important ECM task.

Recently I have been working with the SharePoint 2010 Hold and eDiscovery feature and the feature left me wanting more. Unfortunately, the feature is not easy to use. For example, it was very difficult to construct an effective keyword search. First, the user must have an intimate knowledge of the SharePoint keyword syntax, Secondly, the text box used to enter the keyword search could only show the first 25 characters. Constructing complex keyword searches within this small of a space was impossible.

Getting search administrators to adopt this feature would be difficult given these problems.  Fortunately, SharePoint offers the Search Center and the Advanced Search Web Part that can enhance the Hold and eDiscovery feature.

In this post I will show you how to add a “Hold and Discover” tab to a enterprise search center. This tab will host an Advanced Search Web Part which is an excellent tool for users to easily construct complex keyword searches. The Advanced Search Web Part will post its keyword search to the SearchAndAddToHold.aspx page which is the page that is used for creating Holds. The keyword search then can be displayed in the text box and used for creating Holds. Finally, I will show you how to add a custom button to launch the Text Editor Web Page Dialog to display the complete keyword search and also enable users to construct keyword searches in a larger text space.

Create an Advanced Search Page

This step assumes you have already created a enterprise search center. The search center gives users a consistent user interface for different types of searches. In this case were creating a Hold and Discover type search. To create a new Advanced Search Page in the search center navigate to your search center. From the Site Actions menu select More Options.

The More Options gives you the option to create a publishing page. The Search Center framework contains four types of publishing pages, Advanced Search, Search Results, Search Box and People Search. In the Create dialog select Publishing Page and then click Create. Enter in HoldQueryBuilder as the title and URL, then select the Advanced Search page layout. Then click Create.


Add a new Hold and Discover tab to your search center

Your next step is to add a new tab to your search center. Click on the Site Actions menu and select the Edit Page menu item. Here you will see a Add New Tab link. Click on this link to add a new tab.

Enter in the Tab Page Name text box  “Hold and Discover”. Enter in the HoldQueryBuilder.aspx as the page that will be displayed when a user clicks on the tab, this is the Advanced Search Page created in the previous step.


Customize the Advanced Search Web Part

The next step is to customize the Advanced Search Web Part on the HoldQueryBuilder.aspx advanced search page. Navigate to the advanced search page located at http://severname/yoursearchcenter/pages/HoldQueryBuilder.aspx. Select the Site Actions menu and then select Edit Page menu item. Select the Edit Web Part menu item in the upper right hand corner context menu. In the web part tool pane expand the Search Box section. In the Search Box Section Label enter “Hold and Discover Documents that have…”.  Next, expand the Miscellaneous section. In the Results URL enter in the URL to the SearchAndAddToHold.aspx page.  This is the page for Holds and eDiscovery and is located in the _layouts directory, for example, http://servername/_layouts/SearchAndAddToHold.aspx. The Results URL is where the Advanced Search Web Part will send the constructed keyword search  using a query string appended to this URL. Click OK when done. Finally in the Page tab, click the Check In ribbon button, then in the Publish tab, click the Publish ribbon button.


Almost Finished

So below is what you have so far. You have a search center with a new tab called “Hold and Discover”. When a user clicks on it they are presented with the customized Advanced Search Web Part. After filling in search criteria and clicking the search button the user is presented the SearchAndAddToHold.aspx.

Hooking up it all up

When you click on the search button the Advanced Search Web Part appends the keyword search as a querystring argument to the SearchAndAddToHold.aspx page and navigates to it. Now we need to inject some java script into the SearchAndAddToHold.aspx page to grab the keyword search, URL decode it, and insert it into the search criteria text box. Below is a screen shot showing how the keyword search is displayed after the injection.

To capture the querystring and place the keyword search in the search criteria text box, the following java script should be inserted at the bottom of the SearchAndAddToHold.aspx page.

<script type="text/javascript">

    var keywordQuery = querySt("k");

    if(keywordQuery != undefined)
        document.getElementById('<%=m_tbSearchString.ClientID%>').value = decodeURIComponent (keywordQuery);

    function querySt(ji) {
        hu =;
        gy = hu.split("&");

        for (i = 0; i < gy.length; i++) {
            ft = gy[i].split("=");

            if (ft[0] == ji) {
                return ft[1];

    function loadViewer(builderUrl, editorId, dialogFeatures) {
        var pReturnValue = showModalDialog(builderUrl, editorId, dialogFeatures);
        editorId.value = pReturnValue;
        try {
        catch (exception) {


This java script code is executed when the page loads. It calls the “querySt” function to grab the value of the “k” querystring variable which contains the keyword search built by the Advanced Search Web Part. If the value is present a jquery function decodes it since it is URL encoded. The java script then takes the decoded value and places it in the search criteria text box.

Something is missing

You can now leverage the Advanced Search Web Part to easily construct complex keyword searches without having to know the keyword search syntax. As you can see the keyword search built in the example cannot be completely viewed by the user. It would be nice to view the complete query. This would help the user to learn the syntax or even add additional conditions. Many SharePoint web parts use the Text Editor Web Page Dialog. For example, the Core Search Results Web Part uses it to help users edit xml to add additional return properties to display. The  following html can be added the the SearchAndAddToHold.aspx page to display a button next to the search criteria text box. This will call the “loadViewer” java script function listed in the previous code.  This function launches the Text Editor with the keyword search shown. The user can now view the complete keyword search and even edit it.




It is important to remember that only site collection administrators and users given permission to the Holds list can access the SearchAndAddToHold.aspx page. If the user does not have this permission then when clicking the search button an “Access Denied” page will be displayed.  In addition, you should create a backup copy of the SearchAndAddToHold.aspx file located in the layouts directory. You could even make a copy, rename it, make changes, and have the Advanced Search Web Part use the new web page.

This post has shown how to create a useful Hold and eDiscovery search. The Advanced Search Web Part is a great tool to create complex keyword searches without having to know the syntax. The actual search then can be viewed or edited easily using the familiar Text Editor used through out SharePoint. The Hold and eDiscovery process is an important process in an ECM system. Should it not be easy and useful to use? You can use this example as a starting point to help record administrators be more productive.

Wednesday, June 1, 2011

The Life and Times of a SharePoint Search Results Click Event

Technorati Tags: ,,

In my previous posting I wrote about how to customize your own ranking models. Ranking models are used to sort search results according to their relevance. The higher the rank of the document the higher it is listed in the search results.  SharePoint Search 2010 implements the BM25 ranking model. This model uses field weighting (managed properties) for dynamic ranking. For instance, some properties are more important than others, like title or social rating.  For static ranking calculations, SharePoint search ranks a document higher  if a document in a search results set was visited (click through) from a search results page.  In this post I will explain how SharePoint Search implements the tracking of click through events by users from the search results page. Also, I will show how custom search applications can leverage a SharePoint web service and the object model to log click through events.


Enable Query Logging

The first step of making sure search result click through events are used in ranking calculations is to enable query logging in the Search Service Application associated with the web application.  This can be enabled by navigating to the Search Service Application in Central Administration and clicking the Enable link next to Query Logging.



The Life of a Search Results Click Event

The flow diagram below illustrates the steps taken to process a search result click through in order for it to affect the relevance ranking of the document.


The first step of this process is initiated by the Core Search Results Web Part using JavaScript. The web part emits JavaScript to call the Search web service’s RecordClick method when a user clicks on the title or the URL of the result item. You can easily verify this using Fiddler. In the second step the Search web service calls the associated Search Service Application’s RecordClick method. The service application then calls the internal QueryLogger’s RecordClick method which in turn then calls the internal QueryLogQueue. This object caches the information sent and periodically flushes the data to the MSSQLogUnprocessed table contained in the Search Service Application’s database. These unprocessed URLs are processed and used in the next ranking calculation.

In addition to tracking click through events from search result items the web part will also track click through events for Best Bets. Best Bets are suggested links set up by search administrators that represent the most relevant documents for given keyword terms. Best Bet click through events are used to help manage Best Bet using the Best Bet Usage page in the Search Keywords section of Site Collection Administration. The search administrator can use this information to determine if Best Bets are being used.


Enable Custom Search Applications Click Through

SharePoint Search only records the search result click through event if query logging is enabled and if the search results are hosted in the Core Search Results Web Part. Custom search applications that do not use this web part must implement it. There are two methods to track search result  click through events, one using the Search web service’s RecordClick method or the SearchServiceApplication class’s  RecordClick method.


Search Web Service RecordClick Method

Custom search applications running remotely should use the Search web service to record a click through event. The important task is to create and populate the xml that is sent to this method. The xml represents a serialized Microsoft.Office.Server.Search.Query.QueryInfo class. If you reflect the QueryInfo class you will see custom xml serialization attributes and elements defined for each property. The attributes are cryptic. Below is an example of an xml serialized QueryInfo object:

The “c” element represents the URL that was clicked. The “g” attribute represents the guid ID for the SPSite that the search was executed from and is required. The “qi”  element is required and represents the id of the query. In the case of custom search applications you can create your own guid for the query id. Finally, the “t” attribute represents the time that the search was executed and is required. You must also send it in UTC format. Below is sample code for recording a click through event using the Search web service.

public static void RecordSearchClick()
    string xmlTemplate = "<i  ";
    xmlTemplate += "g=\"e1c43a36-4dcf-4fd9-9287-30e7773c4159\" ";
    xmlTemplate += "t=\"2011-05-31T21:49:28.0642745-05:00\" ";
    xmlTemplate += "xmlns=\"urn:Microsoft.Search\"> ";
    xmlTemplate += "<qi>" + Guid.NewGuid().ToString() + "</qi>";
    xmlTemplate += "<c>http://basesmc2008/tester/pdfimg.pdf</c> "; 
    xmlTemplate += "</i>";

    search.QueryService queryService = new search.QueryService();
    queryService.Url = "http://basesmc2008/_vti_bin/search.asmx";
    queryService.UseDefaultCredentials = true;



Search Service Application RecordClick Method

Custom search applications that can access the SharePoint object model should use the Microsoft.Office.Server.Search.Administration.SearchServiceApplication object associated with the web application it is running in.  This class exposes a RecordClick method using a QueryInfo object as an argument.  The Search web service RecordClick method calls this same method by de-serializing the xml sent to it into a QueryInfo object, then passing it as an argument. The code below shows you an example.

SPFarm farm = SPFarm.Local;
SearchServiceApplication searchApp = (SearchServiceApplication)farm.Services.
    Applications.GetValue<SearchServiceApplication>("Search Service Application");

using(SPSite site = new SPSite("http://basesmc2008"))
    QueryInfo qi = new QueryInfo();
    qi.QueryGuid = Guid.NewGuid().ToString();
    qi.ClickedUrl = "http://basesmc2008/tester/pdfimg.pdf";
    qi.SearchTime = DateTime.Now;
    qi.ClickTime = DateTime.Now;
    qi.SiteGuid = site.ID.ToString();


Relevant Clicking

Search result click through events influence the relevance of a document. The action is important enough for Microsoft to integrate the tracking of this action directly into the Core Search Results Web Part. If your custom search applications do not extend this web part, then you can enable the same type of tracking using one of these two methods. The QueryInfo class is the structure used to hold this information. I recommend looking at this class further. The class has many properties and is also used for logging queries for search administration reporting. Other properties may play a role in influencing relevance, for example, the NonClickedUrls property can hold a string array of URLs that were not clicked. So it appears that not clicking on search result items can lower their relevance.