Microsoft has recently released a new office 365 service called SharePoint Syntex. SharePoint Syntex adds Artificial Intelligence (AI) capabilities to SharePoint document libraries. SharePoint Syntex is the first product to be released from Microsoft project Cortex. SharePoint Syntex provides us with out of the box capabilities to build no-code AI models and apply them to document libraries. SharePoint Syntex is now available for Office 365 commercial customers. You can integrate this service into your Office 365 tenant. There is also a free trial available for one month.

SharePoint contains collections of documents with hidden knowledge inside. SharePoint Syntex helps us extract information from these documents that is important to us using built-in AI models. We can use this metadata to process the documents automatically. Since we do not have to extract the document’s metadata anymore manually, it saves time and money. There is no need to call the external APIs anymore to extract the metadata from document libraries’ documents; SharePoint Syntex brings us two no-code AI models: The Document Understanding model and the Form Processing Model. Below is the example whiteboard diagram of SharePoint Syntex high-level explanation.

AI model Diagram

Document Understanding Models:

Document understanding models work best with unstructured documents (which have more text) such as contracts, resumes, letters, email messages, health records. You can create a document understanding model using the SharePoint Content Center site (Model creation interface) and then apply it to SharePoint document libraries in your Office 365 tenant. The document understanding model comes in two models:

  • Classifier: Identify and classify the documents (Examples: resumes, contracts, letters) uploaded to document libraries.
  • Extractor: Optionally add the extractor to your model to extract the keywords from essential documents. For example, you may extract critical metadata from your document like “Person Name,” “Organization Name” “Contract Start Date” and add them as column values in a document library, and you can use these metadata to process your document further automatically. Example: If “Contract Start Date” is more significant than today, send an email to a group of people.

To train the Classifier and Extractor, documents must have common text to identify using phrases or patterns. Example: If we want to create a model’s explanation for “Resumes,” we can add phrases like “Name,” “Degree,” “Address,” “Programmer,” “Network Engineer,” “Software Engineer” in the model explanation phrase list, which helps the model to identify the document type as “Resume.” Beyond the classification and extraction of keywords, you can also apply a retention label to document understanding models. These cannot be deleted from the document library for the specified time period. Below diagram shows the key steps we can perform from the SharePoint Content Center Site:

Figure 2 Document Understanding

Form Processing Model

The form processing model works best with structured documents like forms, purchase orders, and invoices. Unlike the document understanding model, Form processing can be created directly from SharePoint document libraries, using Microsoft PowerApps AI Builder and Flow. Using the Form Processing model, we can extract the key-value pairs and table data from structure or semi-structured documents and add them as column values in SharePoint libraries.

The below diagram shows the key steps to create and publish the Form Processing model.

Create a Form Processing Model

SharePoint Library Form ProcessingConclusion

SharePoint Syntex can help organizations to automate the business processes since it automatically extracts metadata (information) from the documents. We can use metadata to process documents further, using Flow or any other workflow tools. Now, we have meaningful metadata available on document libraries, which improves the search results. It helps manage the compliance risk since the AI model can also apply a retention label to documents, not deleted from libraries for a specific period. We are getting these features with no code AI Models! This year, Microsoft Project Cortex is also coming up with many new AI features (Topic cards, Topic Pages, Knowledge Centers) in Office 365, which will automatically help us turn content into knowledge.

Referenced for getting started

Recently we integrated Azure QnA Maker Service with SharePoint Search page for one of our DoD Customers so thought to share my experience and lesson learned with broader audiences. Generally, QnA Maker is used when designing a Chatbot, but, according to customer requirements, we integrated with an existing search page, so users do not need to go to multiple places for information. A single place to search all the contents, including static Questions and Answers (FAQs).

Introduction

Azure QnA Maker is one of the Azure Cognitive Services. It is a PaaS service, so no need to maintain any infrastructure. It is available within Azure Gov at DoD Impact Level (IL) 2 and can be configured to store the response data at higher DoD impact levels.

QnA Maker consists of two main parts, QnA Maker Portal site, which is a very user-friendly web interface to import questions and answers, and QnA Maker backend Azure service resources.

Use Case: If your organization has static questions and answers sets in word documents, PDFs, on web pages, etc., you can use the Azure QnA Maker portal to quickly import these questions and answers into QnA Maker to create knowledge bases. It can also automatically extract questions and answers sets from semi-structured contents, manuals, guidelines, etc. Once Questions and Answers are in QnA Maker, your application (Examples: Custom Bot or Custom Search Page) can call QnA Maker Rest API to answer the user’s question.

Asking questions through custom chatbot

High-Level Architecture

It is necessary to understand the QnA Maker architecture if you are working with Government clients and planning to upload answers in QnA Maker, then consider hardening the backend resources of Azure QnA Maker. Apply Network settings on App Service and Search Service. Check out this reference from Microsoft covering backend resources.

High-Level Architecture

Decision Making Between LUIS and QnA Maker Service

It is not hard to decide whether we should use LUIS or QnA Maker to design the Chatbot or any custom applications?

When to use the QnA Maker?
If your organization has lots of static questions and answers ( Example: Questions/Answers Sample), take advantage of out of the box features of QnA Maker. Upload your static questions and answers into QnA Maker Portal, and your application can call QnA API to search for questions asked by the user on the front end application and return the response.

When to use the LUIS?
If you would like to design the application which needs to extract the information from the user’s questions and further process their intents, then use the LUIS.

Example Application: I want to design Bot, where users can ask Bot to create various application development projects. Below are 2 example requests:

Hi, can you create .net core application in C:/Code folder?
Hi , can you create the node.js application in C:/Projects/Code folder?

In this case, LUIS API can extract the word “.net core,” “node.js,” “C:/Code”, and “C:/Projects/Code “ from the user requests, and our application can further process extracted data and can create the respective projects. See the below screenshot from LUIS output. Base on intent = CreateApplicationProject and AppType = node.js, the application gets enough data for the next steps.
LUIS Output Screen Shot

Lessons Learned

Below are the few lessons learned and essential points about the QnA Maker service.

  • You can easily use Postman to test various rest APIs of QnA.
  • Azure QnA Maker Permission can’t be set from Azure QnA Maker Portal. If you would like to share a knowledge base with multiple users, you must give permission on back-end resources using Azure role-based access control (Azure RBAC). You can create a group ( Example: QnAAdministrators) in Azure AD and grant access to the resource group (Resource group with all QnA Maker resources) by assigning a contributor role to a QnAAdministrator at this scope.
  • You can rename your knowledge base from QnA Portal anytime; it does not change the Knowledgebase Key or Endpoint Key.
  • Take advantage of the Metadata Tag feature, so you do not need to design the separate knowledge bases for all departments.
  • You can add a tag with each question.
  • QnA Maker works best if you import both questions and answers into the knowledgebase. If you would like to search only on questions, use the RankerType=QuestionOnly in the POST body of the GenerateAnswer request.
  • To turn on the Application Insight‘s telemetry (Logging), if you haven’t turned it on during QnA Maker Service creation time, you can turn it on afterward on back-end App Service Resource.
  • Application Insight telemetry is very useful for tracking all the unanswered questions and does not need to write custom code for it. Run the below query to view all unanswered questions.
requests
| where url endswith "generateAnswer"
| project timestamp, id, url
| parse kind = regex url with *"(?i)knowledgebases/"KbId"/generateAnswer"
| join kind= inner (
traces | extend id = operation_ParentId
) on id
| extend question = tostring(customDimensions['Question'])
| extend answer = tostring(customDimensions['Answer'])
| extend score = tostring(customDimensions['Score'])
| where  score  == "0" and message == "QnAMaker GenerateAnswer"
| project timestamp, KbId, question, answer, score
| order  by timestamp  desc
  • If you are using Train API Post call to send alternative phrases, it takes 30 to 40 mins to show up on QnA Portal. Once alternative phrases show up on Portal, you need to train and publish the knowledge base again. For Train API to work, make sure you turn on the Active Learning from QnA Portal Settings.
  • QnA Maker has a batch testing tool which is an exe file. Still, you cannot run it from your production environment desktops where an exe file upload might be blocked. In this case, we can easily write custom front end node.js code to read the various questions from the CSV file or Excel file, and for each question, call the GenerateAnswer API to get the answer and score and write it back to the output excel file.

Conclusion

A QnA maker’s beauty is the features of importing your questions and answers sets into QnA Maker from various files like Word, PDF, Text, Excel, etc. You can also bring problems and solutions from published web pages. Once your knowledgebases are in QnA Maker, we can quickly design questions/answers Bot applications without adding many custom codes. QnA Maker is smart enough to search your full question text or phrases from your questions and answers. It can also understand the spelling mistakes and returns the correct answer.

QnA Maker is not only about the QnA Maker Portal site. It does have back-end resources created in your Azure Subscription. If you plan to use QnA maker for larger knowledge bases, consider increasing the SKU of QnA Maker back-end resources and considering hardening the security of back-end resources.

If you plan to design the application that needs to extract the information from the user’s question and further process it, consider using the LUIS.

I support projects where we have platforms like SharePoint, are looking towards adopting PowerApps, and have Azure Government subscriptions. While learning about containers, I wondered where they could fit into an environment where many applications are developed on top of SharePoint. This helped me better understand containers and discover the perfect use case. I thought of sharing my architecture idea through this blog post. If you would like to learn more about container architecture, read this blog post on Container Architecture Basics from our VP of Solution Engineering, Brent Wodicka.

Architecture Idea

If you are maintaining a SharePoint on-premise farm in a local data center or in the cloud, you cannot replace your SharePoint farm VMs with Containers. But what can you do is take the heavy business logic code out of your custom applications from SharePoint farm and deploy it as services using containers. Deploy your backend services using containers. You can call these services endpoints (Rest API Endpoints) from your SharePoint framework web parts and render UI inside the SharePoint framework webparts.

If your application is developed using PowerApps, I believe you can call custom service endpoints in PowerApps. So, your front end can still be deployed using SharePoint or PowerApps, but your business logic can be deployed as services inside the containers.

Where can containers fit in a multi-platform environment?

Answer: Below is two diagrams that explain the use case of the container in a multi-platform environment discussed above.
Business Logic Services Snapshot

Backend Databases and Cloud Services

Advantages of this Architecture

  • Since you have server-side code deployed inside the Azure Cloud, you can easily integrate with other Azure PaaS services, including Azure SQL Database, Azure Cognitive Search.
  • Since most business logic will deploy inside the containers as services, it is easy to implement these services in any other cloud provider.
  • Suppose you have complex legacy applications developed inside the SharePoint, and you are storing the data in SharePoint lists. In that case, you can move those SP lists data to the Azure SQL database and call the Azure SQL Apis from your services that deployed side the containers. ( See the second diagram above)

Conclusion

Suppose you have heavy business logic written in front end JavaScript files in SharePoint. You can rewrite those codes as server-side code using C# and deploy as services inside the containers and call these services endpoints from SharePoint webparts. Complex application data can even move from SharePoint lists to Azure SQL Databases. Containers solve deploying your custom code as services, but they cannot replace your SharePoint infrastructure.

The first time I was introduced to Azure Cognitive Search was from the Microsoft AI Dev Intersection conference in 2019. I thought to write a quick blog post on it to help others understand its features and benefits. This blog is not only for developers, so if you are a Business Analyst, SharePoint Analyst, Project Manager, or Ops Engineer, you will still find the information useful from this blog.

Search-as-a-Service

Azure Cognitive Search (ACS) is a technique for using artificial intelligence (AI) to extract additional metadata from images, blobs, and other unstructured data. It works well for both structured and unstructured data. In the past, we needed to set up a separate search farm to fulfill the search requirements for a web application. Since ACS is a Microsoft Cloud service, we do not need to set up any servers or be a search expert. You can prove these concepts in front of your customer in minutes.

When can we use it?

Most of the businesses have many handwritten documents, forms, emails, PowerPoints, Word documents, of unstructured data. For handwritten documents, even if you scan and digitize it, how can we make content searchable?  If you have images, drawings, and picture data, how do we extract text contents out of it and make it searchable? If you have many handwritten documents, you can scan it, upload it to Azure Blob Storage containers in an organized fashion and Azure Cognitive search can import the documents from Blob Containers and create the search indexes.  The below diagram shows the paper document flow.

Paper Documents Flow:

Paper Documents Flow

Below are a few cases where ACS can really come handy:

  • If the local-file share has many documents and running out of space. Example: If your organization is storing documents in File Server, you can index those documents using ACS and can provide a good search experience so users do not have to use Windows, search explorer to search. You can design nice web application UI which can search using ACS indexes.
  • The customer already has data in the cloud. Like data stored in Azure Blob Storage, Azure SQL Database, or Azure Cosmos Db. ACS can easily connect and create indexes on Azure Blob Storage, Azure SQL Db, and Azure Cosmos DB.
  • International business companies have documents in many languages. Out of the box, ACS search indexes translated results in many different languages. You can show your search result in a different language as well.
  • The client needs to apply AI to business documents.
  • Documents are lacking the Metadata. Example: Documents that are having Title only as metadata so all you can search by is Title! But ACS can extract many key phrases from documents, and we can search on key phrases as well.

We will next learn how to quickly prove this concept.

Creating Service and Indexes from Azure Portal

The below diagram shows the simple flow from the Azure portal. You can prove the ACS concepts in front of clients in minutes.

Creating Service and Indexes from Azure Portal

Log in to the Azure portal and create the Azure cognitive search service. You can find steps on how to create ACS here.

Once your service has been created, follow the below steps to quickly prove the concept.

  • Step 1: Start with documents (unstructured text) such as PDF, HTML, DOCX, Emails, and PPTX in Azure Blob storage. Upload your contents in Azure blob Storage and in ACS. Import your data from Azure Blob Storage.
  • Step 2: Select this option if you would like to apply cognitive skills (see the next section for understanding the cognitive skills)
  • Step 3: Define an index (structure) to store the output (raw content, Step 2-generated name-value pairs).
  • Step 4: Create an indexer, Indexer fills the data into your index fields.

(See the next section for understanding the Index and Indexer)

  • Step 5: You can quickly search on indexes by using Azure Search Explorer.

Understanding Index and Indexer

The search index is like you are creating an empty table and fields. If you want to search on your data, first we need to figure out which fields we want to make it searchable. Once we decide the fields, how can we populate data into it? The search indexer pulls the data from your source and fills your search indexes with data so you can search on search indexes. It is very quick to define your search indexes and create an indexer from Azure Portal in ACS. In ACS search index is just Json objects.

Understanding Index and Indexer

Understanding Text Cognitive Skills and Image Skills 

Out of the box Text Cognitive skills in ACS can extract the people’s names, organization names, location names, and key phrases from your data or documents. Text Cognitive skills can also translate the result in different languages and can also detect the language.

See below an example of results translated into the Hindi language.

Understanding Text Cognitive Skills and Image Skills 

Image skills can generate tags and captions from images and can also identify celebrities.

See below JSON search index as an example of Image cognitive skill.

Image Cognitive Skill

Conclusion:

Since Azure Cognitive Search is cloud service, it is very quick to use it if you already have data in cloud or on-premises. If you have data in your own data center, you can push the data into Azure cognitive search indexes. Below two are my favorite demo sites, they used ACS to extract the content out of paper documents and images.