What’s New in SharePoint 2013 Search (A Developer’s Perspective) Part Two

Technorati Tags: SP2013,Search,ECM

In my first post of this series Part One I talked about how SharePoint 13 preview search has made changes to the API to accommodate the ability to store search objects at different levels. These levels are the Search Service Application, Site Collection and the Site (Tennant). One of these objects is the new result source. It is a combination of SP2010 Federated Locations and Scopes. The result source is a sophisticated tool to allow administrators to construct fine tuned sources for user queries. The sources can be SharePoint (local and remote), file shares, Microsoft Exchange or even custom sources. In this post I will show you how to use the new SearchExecutor class and search using  FQL (FAST Query Language) with it. I will also show you how to leverage the result sources to execute multiple queries and bring back results from different sources in one method call. Please note this posting is based on features in SharePoint 2013 Preview and are subject to change.

The new SearchExecutor class

In SP2010 you used either the KeywordQuery or FullTextSQLQuery classes to execute searches. Since the FAST search technology has now been assimilated into SharePoint 2013  the FullTextSQL syntax is no longer supported. Strangely, the FullTextSQLQuery class still exists and is not marked as deprecated, but it does not work. The KeywordQuery class appears to be the only class to execute queries with. This class uses the new KQL (Keyword Query Language) which is a mixture of SP2010 keyword and FQL syntax. Building Search Queries in SharePoint 2013.

Below is an example of doing a keyword search in SP2010. Notice that the search was invoked by calling the KeywordQuery’s  Execute method

SharePoint 2010 Keyword Search

public static DataTable ExecuteKeywordSearch(string queryText) {     ResultTableCollection rtc = null;     DataTable retResults = new DataTable();     using (SPSite site = new SPSite("http://basesmc2008"))     {         using (KeywordQuery query = new KeywordQuery(site))         {             query.QueryText = queryText;             query.ResultTypes |= ResultType.RelevantResults;             query.KeywordInclusion = KeywordInclusion.AllKeywords;             query.RowLimit = 500;             query.ResultsProvider = SearchProvider.SharepointSearch;             query.SelectProperties.Add("Path");             query.SelectProperties.Add("IsDocument");             rtc = query.Execute();             if (rtc.Count > 0)             {                 using (ResultTable relevantResults = rtc[ResultType.RelevantResults])                     retResults.Load(relevantResults, LoadOption.OverwriteChanges);             }         }     }     return retResults; }

The next example is a SharePoint 2013 keyword search. It uses the new SearchExecutor class and the ExecuteQuery method taking the KeywordQuery object as an argument. You no longer need to set the ResultsProvider property since there is just one provider. Also you do not need to set the ResultTypes property to tell the search engine what type of results you want. Finally, you need to use the new Filter method on the returned ResultTableCollection object to retrieve the results you want. In SP2010 you used an enumeration and the ResultTableCollection’s indexer. The Filter method is much more flexible but you need to hard code string property names rather than using an enumeration. The string argument represents property name of the ResultTableColection, and the second argument is an object value for that property.

SharePoint 2013 Keyword Search

public static DataTable ExecuteKeyWordSearch(string queryText) {     ResultTableCollection rtc = null;     DataTable retResults = new DataTable();     using (SPSite site = new SPSite("http://basesmc15"))     {         using (KeywordQuery query = new KeywordQuery(site))         {             query.QueryText = queryText;             query.KeywordInclusion = KeywordInclusion.AllKeywords;             query.RowLimit = 500;                             query.SelectProperties.Add("Path");             query.SelectProperties.Add("IsDocument");             SearchExecutor se = new SearchExecutor();             rtc = se.ExecuteQuery(query);             if (rtc.Count > 0)             {                 var results = rtc.Filter("TableType", KnownTableTypes.RelevantResults);                 if (results != null && results.Count() == 1)                     retResults.Load(results.First(), LoadOption.OverwriteChanges);             }         }     }     return retResults; }

What about FQL?

SharePoint allegedly supports FQL queries. So if you have legacy solutions that use FQL then these should be supported. You must set the EnableFQL property of the KeywordQuery class to true. Unfortunately, results are inconsistent. The results are returned every 4 searches, the other 3 no results are returned. There is additional set up needed in order for FQL to work.  A bigger issue with using FQL is that FQL searches will not work with any result sources that have Query Transformations. The reason for this is that Query Transformation are additional conditions that are appended to the query text. Many of the out of the box result sources including the default have transformations that use KQL (Keyword Query Language). Therefore when you execute a FQL search with a result source with a Query Transformation you will receive a QueryMalformedException. You must set up a result source without a transformation and use that when executing FQL searches. So FQL support is theoretical at this point of the beta.

The new SearchExecutor is built for executing multiple queries at once

So why the switch to the new SearchExecutor class? The big reason is to give you more flexibility to execute multiple queries either on the server or remotely using the new client object model for search. The code below demonstrates how to create and issue multiple queries to the server at once. The first query uses the default result source, the second query uses a different result source by setting the KeywordQuery’s SourceID property. It uses a method for getting a search result source from part one of this series. The SearchExecutor’s ExecuteQueries method takes a dictionary of strings and KewywordQuery objects. The strings are friendly names you can give the queries so you can refer to them in the returned dictionary of strings and ResultTableCollections. Also notice that you can check the QueryErrors collection on each returned ResultTableCollection. This is useful if you set the handleExceptions argument of the ExecuteQueries method to true. If you set this to false the first error that occurs on any of the queries will halt execution and throw the error to the caller. You should be aware that if you are sending a query to a remote source you must set up a trusted connection.

The code also illustrates the inefficiencies of creating multiple KeywordQuery objects to submit to multiple sources. Typically you would submit the same terms, returned properties, row limits, and other properties. It would be nice if there was a method to clone KeywordQuery objects and then just set the SourceID properties. A good opportunity for an extension method.

 

public static Dictionary<string,DataTable> ExecuteMultipleKeyWordQueries() {     DataTable tableResults = null;     Dictionary<string, DataTable> retResults = new Dictionary<string, DataTable>();     using (SPSite site = new SPSite("http://basesmc15"))     {         Dictionary<string, Query> queries = new Dictionary<string, Query>();         Dictionary<string, ResultTableCollection> results;         using (KeywordQuery query1 = new KeywordQuery(site))         {             query1.QueryText = "title:\"Viewer Testing with PDF\"";             query1.RowLimit = 500;             query1.SelectProperties.Clear();             query1.SelectProperties.Add("Path");             query1.SelectProperties.Add("IsDocument");             query1.SelectProperties.Add("Title");             SearchExecutor se = new SearchExecutor();             queries.Add("local", query1);             using (KeywordQuery query2 = new KeywordQuery(site))             {                 query2.QueryText = "title:\"Viewer Testing with PDF\"";                 query2.RowLimit = 500;                 query2.SelectProperties.Clear();                 query2.SelectProperties.Add("Path");                 query2.SelectProperties.Add("IsDocument");                 query2.SelectProperties.Add("Title");                 Source remoteSource = GetSearchSource("remoteFarm");                 query2.SourceId = remoteSource.Id;                 queries.Add("remote", query2);                 results = se.ExecuteQueries(queries, true);                 foreach (KeyValuePair<string,ResultTableCollection> result in results)                 {                     if (result.Value.Count > 0 && result.Value.QueryErrors.Count() == 0)                     {                         var resultTable = result.Value.Filter("TableType", KnownTableTypes.RelevantResults);                         if (resultTable != null && resultTable.Count() == 1)                         {                             tableResults = new DataTable();                             tableResults.Load(resultTable.First(), LoadOption.OverwriteChanges);                             retResults.Add(result.Key, tableResults);                         }                     }                 }             }         }     }     return retResults; }

 

It is all about Federation

The SearchExecutor is built for Federation of different search sources. Federation is key piece in making SharePoint search more scalable by eliminating the need to crawl and index other searchable sources. Designing the SearchExecutor  to easily execute and process the results of multiple queries, even FQL queries, helps you leverage Federation in your solutions. Even the CSOM has exposed this feature in the Microsoft.SharePoint.Client.Search assembly and namespace. Retrieving results is different in the CSOM since there is no Filter method exposed on the ResultsTableCollection due to the nature of the execution. You just use basic LINQ to get the appropriate DataTable. You must call the SearchExecutor’s ExecuteQuery method and then call the Context’s ExecuteQuery method. This looks strange but if you have used the CSOM in the past it makes sense.

Client Object Model Keyword Search

public static DataTable ExecuteKeywordQuery(string query) {     DataTable retResults = new DataTable();     ClientContext con = new ClientContext("http://basesmc15");     KeywordQuery kq = new KeywordQuery(con);     kq.QueryText = query;     kq.SelectProperties.Add("testcol");     SearchExecutor se = new SearchExecutor(con);     var results = se.ExecuteQuery(kq);     con.Load(kq);     con.Load(se);     con.ExecuteQuery();     var result = from r in results.Value where r.TableType == KnownTableTypes.RelevantResults select r;     if (result != null)     {         //get column names         foreach (var col in result.First().ResultRows.First())         {             retResults.Columns.Add(col.Key.ToString(), col.Value != null ? col.Value.GetType() : typeof(object));         }         foreach (var row in result.First().ResultRows)         {             retResults.LoadDataRow(row.Values.ToArray(), LoadOption.Upsert);         }         retResults.AcceptChanges();     }     return retResults; }

SharePoint 2013 search has many new features. I will be posting more about these soon.

What's New in SharePoint 2013 Search (A Developers Perspective) Part One

Technorati Tags: SP2013,Search,ECM

Microsoft has made dramatic changes to serch in SharePoint 2013 Preview. There have been so many changes it is impossible to cover them all in one post. In this post I am going to talk about some of the changes you need to know about as a SharePoint Search developer. I will cover how the API has changed to accommodate the importance of Office 365. Secondly, I will show you how search scopes have been eliminated and replaced with result sources. Please remember that the findings in this post are based on the SharePoint 2013 Preview and are subject to change.

What happened to my Search Scopes?

In SharePoint 2013 search scopes are deprecated. Basically they are gone and not being used. You can access two system scope “All Sites” and “People” from the search service  application result sources page. However, you cannot create new search scopes nor can they be displayed in a dropdown list next to the standard search box. This is unfortunate because scopes could help users  find what they are looking for easier. SharePoint 2013 has replaced scopes with result sources. Result sources are SharePoint 2010 Federated Locations and Scopes combined with an easier way to define scoping rules.

Result sources make it easier to search remote sources including other SharePoint farms. Searching of remote SharePoint farms is easier than before because SharePoint 2013 relies on claims based authentication. No longer do you have to publish and subscribe to a remote service application and designate certain accounts for searching. Users can connect to the source using the default SharePoint authentication.

 

With SharePoint 2010 scopes you defined different types of rules which were limited to text properties and web addresses. In SharePoint 2013 there is a new query builder to help build a “Query Transformation”. You can now define any type of managed property inclusion or exclusion. The builder also gives you a large set of built in options to select from to help you build complex transformations, for example the name of the user who is running the query or a placeholder for a token from the request URL. Also, the query builder allows you to select from a list of all the managed properties and include property conditions similar to what you used in Advanced Search in SharePoint 2010. Here is an example of query text built by the builder to include content that has the user’s name running the query, a token from the URL and a ContentTypeID that starts with a certain value:

{searchTerms}{User.Name} {URLToken.1} ContentType=0x01*

You must include {searchTerms} to have the conditions appended to the query being submitted. The new query builder allows you to define much more complex conditions than the old search scoping rules. The builder also allows you to include custom sorting and gives you the ability to test the transformation and view the results it produces. There should be much more documentation on the capabilities of this builder and its syntax forthcoming from Microsoft.

 

 

How to use the new Result Source with your query

In SharePoint 2010 you appended scope names to your query text. In SharePoint 2013 you must set the KeywordQuery classes’ SourceID property to the GUID from a Microsoft.Office.Server.Search.Administration.Source. If you do not set this property to the source you want it will use the default source defined from the search service application. There seems to be no way from the UI to set the default result source for a search service application. So if you do not want to use the default result source you must retrieve the one you want. SharePoint 2013 is all about Office 365 and you can see evidence of this by the fact that you can define search managed properties, result sources, query rules and result types down to the site level. The SharePoint 2013 search API needed a way to retrieve these search objects easily, so the SearchObjectLevel and SearchObjectOwner classes were introduced to accomplish this. These two classes can be passed into various method calls to retrieve search objects, or used to create a SearchObjectFilter object to pass to the methods.

The SearchObjectFilter (Owners or Levels)

The SearchObjectFilter class is used in most cases to filter out the type of search objects you want returned. The SearchObjectFilter has two constructors one which takes a SearchObjectLevel enumeration and one that takes a SearchObjectOwner.

The SearchObjectLevel enumeration has the following levels:

public enum SearchObjectLevel  {         SPWeb = 0,         SPSite = 1,         SPSiteSubscription = 2,         Ssa = 3,  }

Using this enumeration in the constructor of the SearchObjectFilter you can filter based on Ssa = Search service application, SPSite = Site Collection, SPWeb = Site, and SPSiteSubscription = tenant.  Below is a code sample that uses a SearchObjectFilter based on a SearchObjectLevel to return a List of SourceRecords defined at the site collection level.

public static List<SourceRecord> GetSearchSourcesUsingLevel() {     using (SPSite site = new SPSite("http://basesmc15"))     {         SPServiceContext serviceContext = SPServiceContext.GetContext(site);         var searchProxy =             serviceContext.GetDefaultProxy(typeof(SearchServiceApplicationProxy))             as SearchServiceApplicationProxy;         SearchObjectFilter filter = new SearchObjectFilter(SearchObjectLevel.SPWeb);         //true is the default         filter.IncludeHigherLevel = false;         //false is the default         filter.IncludeLowerLevels = true;         List<SourceRecord> sources =             searchProxy.GetResultSourceList(filter, false).ToList<SourceRecord>();         return sources;     } }

This code will work only if there is an available SPContext. If you are running this in a console application or a timer job an error will be thrown stating a valid SPWeb could not be found on the SPContext. Hopefully, this will be fixed by release. The recommended approach is to create a SearchObjectOwner object using the appropriate SearchObjectLevel and a SPWeb. Then use the SearchObjectOwner to create the SearchObjectFilter. This enables the internal code to determine the scope of your search correctly.

public static List<SourceRecord> GetSearchSourcesUsingOwner() {     using (SPSite site = new SPSite("http://basesmc15"))     {         using (SPWeb web = site.OpenWeb())         {             SPServiceContext serviceContext = SPServiceContext.GetContext(site);             var searchProxy =                 serviceContext.GetDefaultProxy(typeof(SearchServiceApplicationProxy))                  as SearchServiceApplicationProxy;             SearchObjectOwner so = new SearchObjectOwner(SearchObjectLevel.SPSite, web);             SearchObjectFilter filter = new SearchObjectFilter(so);             //true is the default             filter.IncludeHigherLevel = false;             //false is the default             filter.IncludeLowerLevels = true;             List<SourceRecord> sources =                 searchProxy.GetResultSourceList(filter, false).ToList<SourceRecord>();             return sources;         }     } }

Notice the IncludeHigherLevel and IncludeLowerLevels properties. You can use these to adjust the filter to return sources above and below the SearchObjectLevel used to create the filter. For example if you wanted to return result sources from current site collection and the search service application then you would set the IncludeHigherLevel to true and IncludeLowerLevels to false. Below is example code getting a result source and using it in a query. The code uses the new SearchExecutor class to execute a query.

public static DataTable ExecuteKeyWordQuery() {     ResultTableCollection rtc = null;     DataTable retResults = new DataTable();     using (SPSite site = new SPSite("http://basesmc15"))     {         using (KeywordQuery query = new KeywordQuery(site))         {             query.QueryText = "microsoft isdocument:1";             query.RowLimit = 500;             query.SelectProperties.Clear();             query.SelectProperties.Add("Path");             query.SelectProperties.Add("IsDocument");             query.SelectProperties.Add("Title");             var source = from s in GetSearchSourcesUsingOwner()                          where s.Name == "mysource" select s;             if(source != null)                 query.SourceId = source.First().Id;             SearchExecutor se = new SearchExecutor();             rtc = se.ExecuteQuery(query);             if (rtc.Count > 0)             {                 var results = rtc.Filter("TableType", KnownTableTypes.RelevantResults);                 if (results != null && results.Count() == 1)                     retResults.Load(results.First(), LoadOption.OverwriteChanges);             }         }     }     return retResults; }

 

After result sources have been defined they can be used in any of the available search web parts that support sources. SharePoint 2013 search is focusing more on federation of search results rather than have users choose which scopes they want to search. The focus seems to be to create custom search result pages with multiple types of federated results.

More changes in SharePoint 2013 Search

In my next post I will talk about leveraging result sources and executing multiple queries with different sources and how SharePoint 2013 makes this easy. I will also talk about the changes in search syntax and enabling the use of FQL in your queries.