Microsoft Artificial Intelligence: March 2018

Thursday, March 29, 2018

Building video Indexer xamarin Mobile Application Using Cognitive Service

Introduction:

Microsoft introduced the public preview of Video Indexer as a part of Cognitive Service. Previously, we used Video API but now it's replaced with Video Indexer. Video Indexer automatically extracts the metadata and builds intelligent innovative AI applications based on Video and Audio.

In this article, I will show how to integrate Video Indexer embeddable widgets into the mobile application.

Create Video Indexer Account:

The implementing Video Indexer into the application, you must have to create account and upload your video using my previous article as a reference .

Video Indexer supports embedding two types of widgets into your Mobile application

Cognitive Insights – its Includes all visual insights that were extracted from video indexer.
Player -its enable you to stream the video.

Embedding Video:

I will show below steps for how to get embedding source from video indexer portal

Step 1: Login to Video Indexer Portal using Microsoft, google or Azure Active directory account

Step 2: Select your video, whichever need to integrate to the application.

Step 3: Click the "embed" button that appears below the video.

Step 4: After click on “embed” button, Select the widget you want to embed with the desired options. (player/insights)

Step 6: You can copy the embed codes from embed popups and start implement into your application for Public videos. If you want to embed a Private video, you have to pass an access token in the iframe src attribute as a query string .

Create Xamarin / Web Application:

Let's start with creating a new Xamarin Forms Project using Visual Studio. Open Run - Type “Devenev.Exe” and enter - New Project (Ctrl+Shift+N) - select Blank Xaml App (Xamarin.Forms) template.

It will automatically create multiple projects, like .NET Standard, Android, iOS, and UWP.

Implement Cognitive Insights Widget:

Cognitive insights widget contains all the visual insights that were extracted from the video after the indexing process such as people appearances, top keywords, sentiment analysis, transcript, translate, and search.

Let we start implement into the Xamarin Application.

You can add webview control inside the content page as per below xaml design

<?xml version="1.0" encoding="utf-8" ?>

<ContentPage xmlns="http://xamarin.com/schemas/2014/forms"

xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"

xmlns:local="clr-namespace:VideoIndexer"

:Class="VideoIndexer.MainPage">

</ContentPage>

You can use your cognitive insights widget code from c# code as a htmlwebviewsource . You can specify the query string for whatever widgets need to display ( <url>?widgets=people&title=myInsights)

public MainPage()

{

InitializeComponent();

// var browser = new WebView();

var htmlSource = new HtmlWebViewSource();

htmlSource.Html = @"<html><head><script src='https://breakdown.blob.core.windows.net/public/vb.widgets.mediator.js'></script></head>

</body></html> ";

browser.Source = htmlSource;

}

You can deploy the application, the output look like below

Implement Video Insights Widget:

The video player widget is a customized Media Player that except of providing video streaming, contains extra features such as playback speed and closed captions. Refer below code for video insight wedge implementation

public MainPage()

{

InitializeComponent();

// var browser = new WebView();

var htmlSource = new HtmlWebViewSource();

htmlSource.Html = @"<html><head><script src='https://breakdown.blob.core.windows.net/public/vb.widgets.mediator.js'></script></head>

</body></html> ";

browser.Source = htmlSource;

}

Implement Video and Cognitive Insights Widget:

Copy the player and cognitive insights embed code and also include javascript communication file for communicate with each other widget

var htmlSource = new HtmlWebViewSource();

htmlSource.Html = @"<html><head><script src='https://breakdown.blob.core.windows.net/public/vb.widgets.mediator.js'></script></head>

</body></html> ";

browser.Source = htmlSource;

Summary

In this article, we learned how to sign in to Video Indexer and integrate Video Indexer embeddable widgets into the mobile cross platform application. If you have any questions/feedback/ issues, please write in the comments box.

MSDN Source Code

Getting started with the Video Indexer using Microsoft Cognitive Services

Introduction:

Microsoft introduced the public preview of video indexer as part of cognitive service. previously its Video API now its replaced into Video Indexer. Video Indexer automatically extract metadata and build intelligent innovative application based Video and audio.

In this article, I will share how to sign in to video Indexer and upload your video and extracting the meta data and translation.

Create Account:

The developing video indexer application, you must login or create account using any of one below account Azure active directory, Microsoft Scholl account, LinkedIn, google or Facebook to Microsoft VI.

Upload Video:

We need to upload video into MS video Indexer portal, after login, select Upload and Drag drop your video file or provide video web URL for upload to the portal

Provide basic details about video file name, language and privacy setting and click on upload button .

After uploaded video to the portal, Microsoft VI will do process for analyzing and indexing the video.

Once video indexer is done the analyze, you will get email notification with link of video, short description and people face detection.

You can edit the privacy setting from the portal and Microsoft cognitive service will return following analyze report.

Face identification:

Microsoft AI will help detection of faces in a Video. The faces are matched against a celebrity, it will identify the matched name or user can also edit label faces that do not match the celebrity.

Speech to Text:

The Microsoft Video Indexer has speech to text functionality, this will help user to transcript to spoken language .it will support Tamil, English, Hindi, etc. and also you can edit the text . Video Indexer has the ability to map and understand which speaker spoke which words and when.

Identify Objects:

Video Indexer identify the pre-defined 2000 objects based on video background.

Keyword Extractions:

The meta keyword will help for search the video into the large library. Video Indexer extracts keywords based on the transcript of the spoken words and text recognized by visual text recognizer.

Sentiment analysis:

Video Indexer performs sentiment analysis on the text extracted using speech-to-text and optical character recognition, and provide that information in the form of positive, negative of neutral sentiments, along with timecodes.

Translation:

Video Indexer has the ability to translate the audio transcript from one language to another. Video Indexer will supported following multiple language like Tamil, English, Spanish, etc..

Once Video Indexer is done the processing and analyzing video , you can review , edit ,delete and publish the video into the Microsoft VI Portal .

Summary

In this article, you learned how to sign in to video Indexer and upload the video and extracting the meta data and translation. If you have any questions/feedback/ issues, please write in the comment box.

Building Xamarin Mobile Application with Bing AutoSuggest using Cognitive Services

Introduction:

The Bing Autosuggest API returns a list of suggested queries based on the user enters in the search box. Display the suggestions in the search box's drop-down list. The suggested terms are based on suggested queries that other users have searched on and user intent.

In this article, I will show how to generate Bing Search subscription key and integrate into the Xamarin application

You need to create an Azure account and generate subscription key for implement to the application .

Step 1: Sign in to Azure portal .

Step 2: Create On “+ Create a resource “> Under Azure Marketplace, select AI + Cognitive Services and discover the list of available APIs. > Select “ Bing Search v7 APIs”

Step 3: on the create page, Provide the name, pricing, resource group a click on Create

Step 4: wait for few seconds, After the Cognitive Services account is successfully deployed, click the notification or tile in the dashboard to view the account information. You can copy the Endpoint URL in the Overview section and keys in the Keys section to start making API calls in our Xamarin applications.

Create Xamarin Application with Bing AutoSuggest :

Let's start with creating a new Xamarin Forms Project using Visual Studio. Open Run - Type “Devenev.Exe” and enter - New Project (Ctrl+Shift+N) - select Blank Xaml App (Xamarin.Forms) template.

https://lh4.googleusercontent.com/3700JMuOS2-PsW3d7bcg4tHvikptOS945dr6721F-HhLS4ZpQ6BPlSsWI6xZD3U8Uo4rqnYHu0PHS5PLd0qzdVaNH2dAz5af7o_SVg0IWsaZqGaU4NcmMdNmouwzvFkG1pSBldMhQkL0O8L2LA

It will automatically create multiple projects, like .NET Standard, Android, iOS, and UWP.

Install Newtonsoft.Json :

Bing Autosuggest will return Json object value so make sure you have added the Newtonsoft JSON NuGet Package to your all project. Right Click on Solution > Manage Nuget Package > Install Newtonsoft Json

Install Microsoft.Csharp :

This steps is optional, if you get Error "Microsoft.CSharp.RuntimeBinder.Binder.Convert" not found by the compiler for dynamic type so adding a reference as Microsoft.CSharp to the project , this issue will get resolve .

Design View:

After successfully install above two nuget package. Let start design UI design from Dot Standard /PCL project. In PCL Project > Open MainPage.Xaml page and add design Entry and list view control in Xaml

<?xml version="1.0" encoding="utf-8" ?>

<ContentPage xmlns="http://xamarin.com/schemas/2014/forms"

xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"

xmlns:local="clr-namespace:XamarinBingAutoSuggest"

:Class="XamarinBingAutoSuggest.MainPage">

<ListView x:Name="lstautosug" BackgroundColor="Azure" SelectedItem="{Binding Source={x:Reference entrysearch},

Path=Text}"></ListView>

</StackLayout>

</ContentPage>

Configure the project:

Open the MainPage.xaml.cs file from PCL project > replace your subscription key and End url

private string AutoSuggestionEndPoint = "https://api.cognitive.microsoft.com/bing/v7.0/suggestions";

public HttpClient AutoSuggestionClient

{

get;

set;

}

public MainPage()

{

InitializeComponent();

AutoSuggestionClient = new HttpClient();

AutoSuggestionClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "<Key>");

}

Get and Parse Json Data:

HttpClient class provides a base class for get/Post the HTTP requests/responses from a URL. It is a supported async feature of .NET framework. HttpClient is able to process multiple concurrent requests. The following code showing get all Json data using Bing suggestions API url and Parse the json and binding into the list view for autocomplete

private async void OnTextChangesEvent(object sender, TextChangedEventArgs e)

{

try

{

if(entrysearch!=null)

lstautosug.ItemsSource = await GetAutoSuggestResults(this.entrysearch.Text);

}

catch (HttpRequestException)

{

}

public async Task<List<string>> GetAutoSuggestResults(string query)

{

List<string> suggestions = new List<string>();

string market = "en-US";

var result = await AutoSuggestionClient.GetAsync(string.Format("{0}/?q={1}&mkt={2}", AutoSuggestionEndPoint, WebUtility.UrlEncode(query), market));

result.EnsureSuccessStatusCode();

var json = await result.Content.ReadAsStringAsync();

dynamic data = JObject.Parse(json);

if (data.suggestionGroups != null && data.suggestionGroups.Count > 0 && data.suggestionGroups[0].searchSuggestions != null)

{

for (int i = 0; i < data.suggestionGroups[0].searchSuggestions.Count; i++)

{

suggestions.Add(data.suggestionGroups[0].searchSuggestions[i].displayText.Value);

}

return suggestions;

}

Run the Application:

We have completed the coding now start run the application so you can select the platform iOS,Android , Windows and Click on Run (f5) the application .

You can find the source code at C# Corner attachment and Github XamarinBingAutoSuggest repository as Microsoft-Cognitive-Service.

MSDN Source Code

Summary

In this article, you learned about how to generate Bing Autosuggest subscription key and integrate into the Xamarin application .If you have any questions/ feedback/ issues, please write in the comment box.

Suthahar Blog content is moving to MsDevBuild

Thursday, March 29, 2018

Building video Indexer xamarin Mobile Application Using Cognitive Service

Getting started with the Video Indexer using Microsoft Cognitive Services

Building Xamarin Mobile Application with Bing AutoSuggest using Cognitive Services