Enable Swagger in your .Net core 2.0 application: Step by step guide

swa13

Note – You can find the source code of my sample application here.

If you have ever worked with APIs then you might be familiar with Swagger, if you have not heard about Swagger then this post will help you to know the basic of Swagger and the steps to configure Swagger with yours .Net Core 2.0 application.

First of all, let us see what is Swagger

One liner for Swagger – UI representation of your RESTfull APIs

  • Swagger is a set of rules (in other words, a specification) for a format describing REST APIs.
  • The format is both machine-readable and human-readable.
  • As a result, it can be used to share documentation among product managers, testers and developers, but can also be used by various tools to automate API-related processes.

For example, you have set of APIs and you want a proper documentation for these APIs, you can use Swagger. You can even test the API calls from Swagger and there is lot more can be done using Swagger, have a look here for more details.

In this post, we will see how can we use Swagger with .Net Core 2.0 application.

Step by step guide:

Open Visual Studio 2017 and Create New project – > Select Core Web application:

swa1

Click on OK, it will open new template window with different awesome templates. Select Web API:

swa2

It will create the API project with the structure shown below:

swa3

Let us add Swagger to the middleware. Open Startup.cs class and add below code to the ConfigureService method:

// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc();

services.AddSwaggerGen(c =>
{
 c.SwaggerDoc("v1", new Info
 {
  Version = "v1",
  Title = "My First API",
  Description = "My First ASP.NET Core 2.0 Web API",
  TermsOfService = "None",
  Contact = new Contact() { Name = "Neel Bhatt", Email = "neel.bhatt40@gmail.com", Url = "https://neelbhatt40.wordpress.com/"}
  });
 });
}

Once you add above code, an error will be shown because IServiceCollection does not contain the definition for AddSwaggerGen:

swa4

We will have to add Nuget package – Swashbuckle.AspNetCore to add this support which is used to add Swagger tools for documenting APIs built on ASP.NET Core.

Search Swashbuckle.AspNetCore in Nuget package manager and select the package as shown below:

swa5

Once you install the Nuget package, errors will be gone.

Once this is done, we will enable Swagger UI. Add below code in Configure method of Startup.cs class:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
 if (env.IsDevelopment())
 {
   app.UseDeveloperExceptionPage();
 }

 app.UseMvc();

 app.UseSwagger();
 app.UseSwaggerUI(c =>
 {
   c.SwaggerEndpoint("/swagger/v1/swagger.json", "My API V1");
 });
}

We are almost done and we have already enabled Swagger in the Core 2.0 application.

Last but not the least step, we will change the launch browser setting of the application and will tell the application to launch the Swagger when we run the application.

For these Open properties of the application -> Go to Debug tab -> Write swagger in the Launch browser text box:

swa6

That is it. Just run the application and you will see beautiful colorful Swagger landing page as shown below:

swa7

Manage versions of APIs

You can even manage different versions of your APIs, for example, you have v2 APIs as well and you need to enable that in Swagger then you just need to add a couple of steps.

First add new SwaggerDoc in ConfigureService method and then add new endpoint in Configure method in Startup.cs class as shown below:

swa8

Now just run the application and you can select different versions of your APIs by selecting the value from dropdown list as shown below:

swa9

XML documentation

We all are quite familiar with the XML document of the API, for example, you have hosted your APIs and you want to send the details of APIs to another team then you can simply pass XML document which contains all the details of the API like description, desired input, output etc.

Let us see how we can create XML documents with Swagger in .Net Core 2.0 application:

First of all, we need to enable XML support and once we enable the checkbox, the application will create an XML file in the bin\debug folder.

Open properties – > Open Build tab -> Enable check box XML documentation file as below:

swa10

Once this is done, we need to add below lines of code in the ConfigureService method which will include the comments into the XML file:

var basePath = PlatformServices.Default.Application.ApplicationBasePath;
var xmlPath = Path.Combine(basePath, "NeelSwaggerSampleApplication.xml");
c.IncludeXmlComments(xmlPath);

Now add some comments on any random method as shown below:

swa11

And when you build the solution, XML file would be generated in bin\debug folder. Open the XML file, it will show these comments:

swa12

Have a look at the Github page of Swagger UI if you want more information.

You can find the source code of my sample application here.

Hope it helps.

 

 

 

Advertisements

Create your first bot using FormFlow of Bot framework: Demo Movie Booking Bot

flow9

In my last post, I explained how to create your first bot using Visual Studio 2017 which you can find here.

In this post, we will see how to add selection box(forms) to your bot using Microsoft FormFlow.

As explained in my previous post, you can follow the steps to add Bot application into your Visual Studio.

NOTE – the Whole code is available here.

I will use the same solution I created last time and will add one more class which I will name as MovieBooking.

flow1

In this class, we will add different enum, which will work as a select box. It is a real beauty of Bot framework.

For example, let us create the enum for MovieTheatreLocation:

[Serializable]
public enum MovieTheatreLocation
{
  Pune = 1, Bangalore = 2, Mumbai = 3
}

and when we run this on Bot Emulator, it will look like below:

flow2

Let us add all required code to MovieBooking.cs class:

using Microsoft.Bot.Builder.Dialogs; 
using Microsoft.Bot.Builder.FormFlow; 
using System; 

namespace NeelBotDemo.FormFlow
{ 
 [Serializable] 
 public class MovieBooking 
 { 
 public MovieTheatreLocation movieTheatreLocation; 
 public MovieTheatre movieTheatre; 
 public MovieTypes movieTypes; 
 public ClassTypes classTypes; 
 [Optional] 
 public DoYouNeedAMeal doYouNeedAMeal; 
 public FoodMenu foodMenu; 
 public DateTime? DateOfJurney; 
 [Numeric(1, 5)] 
 public int? NumberOfAdult; 
 public int? NumberOfChild; 

 public static IForm<MovieBooking> BuildForm() 
 { 
 return new FormBuilder<MovieBooking>() 
 .Message("Welcome to the Movie Booking BOT created by Neel.") 
 .OnCompletion(async (context, profileForm) => 
 { 
 var userName = string.Empty; 
 context.UserData.TryGetValue<string>("Name", out userName); 
 // Tell the user that the form is complete 
 await context.PostAsync("Thanks for the confirmation " + userName + ". Your booking is successfull"); 
 }) 
 .Build(); 
 } 
} 

 [Serializable]
 public enum MovieTheatreLocation
 {
 Pune = 1, Bangalore = 2, Mumbai = 3
 }
 [Serializable]
 public enum MovieTheatre
 {
 PVR = 1, INOX = 2, CinePolis = 3
 }
 [Serializable]
 public enum MovieTypes
 {
 Hindi = 1, English = 2, Regional = 3
 }
 [Serializable]
 public enum ClassTypes
 {
 PlatinumClass = 1,
 GoldClass = 2,
 Economy = 3
 }
 [Serializable]
 public enum DoYouNeedAMeal
 {
 Yes = 1,
 No = 2
 }
 [Serializable]
 public enum FoodMenu
 {
 Sandwich = 1,
 Noodles = 2,
 Samosa = 3,
 Cookies = 4,
 Juice = 5,
 Tea = 6,
 Coffee = 7
 }

As you can see we have added different enum.

Important Notes:

  • Make sure you select the appropriate name of Enum because it will be automatically reflected in the bot. For example, if you write the name of Enum as MovieTheatreLocation then the bot will use this as:

Please select a movie theatre location

  • Also, note that the sequence of different selection box will depend on how you declare different Enum in code.

For example in above class you have declared Enum in below sequence:

public MovieTheatreLocation movieTheatreLocation;
 public MovieTheatre movieTheatre;
public MovieTypes movieTypes;
public ClassTypes classTypes;
[Optional]
public DoYouNeedAMeal doYouNeedAMeal;
public FoodMenu foodMenu;
public DateTime? DateOfJurney;
[Numeric(1, 5)]
public int? NumberOfAdult;
public int? NumberOfChild;

So Movie theatre selection box will come first and Number of child box will come last.

Now, we need to create a separate dialog for MovieBooking, so add one class called MovieBotDialog:

flow1

In this class, we will write what bot should behave when the user replies something or when the whole conversation is over.

Add code to MovieBotDialog.cs class:

using NeelBotDemo.FormFlow;
using Microsoft.Bot.Builder.Dialogs;
using Microsoft.Bot.Builder.FormFlow;
using System.Linq;
using System.Text.RegularExpressions;
using System.Threading.Tasks;

namespace NeelBotDemo.Dialogs
{
 public class MovieBotDialog
 {
   public static readonly IDialog<string> dialog = Chain.PostToChain()
   .Select(msg => msg.Text)
   .Switch(
     new RegexCase<IDialog<string>>(new Regex("^hi", RegexOptions.IgnoreCase), (context, text) =>
    {
      return Chain.ContinueWith(new RootDialog(), AfterMyDialogContinue);
    }),
    new DefaultCase<string, IDialog<string>>((context, text) =>
    {
      return Chain.ContinueWith(FormDialog.FromForm(MovieBooking.BuildForm, FormOptions.PromptInStart), AfterMyDialogContinue);
    }))
    .Unwrap()
    .PostToUser();

private async static Task<IDialog<string>> AfterMyDialogContinue(IBotContext context, IAwaitable<object> item)
{
   var token = await item;
   var name = "User";
   context.UserData.TryGetValue<string>("Name", out name);
   return Chain.Return($"Please continue typing if you want to book a movie ticket.");
  }
 }
}

As you can see above, we have mentioned if user types “hi” then we will initiate RootDialog class and in RootDialog class we will write some code to collect the name of the user. RootDialog class will store the name into the context user data, so that the name can be used later in the conversation.

Add code to RootDialog class:

using Microsoft.Bot.Builder.Dialogs;
using Microsoft.Bot.Connector;
using System;
using System.Threading.Tasks;

namespace NeelBotDemo.Dialogs
{
  [Serializable]
  public class RootDialog : IDialog
  {
    public async Task StartAsync(IDialogContext context)
    {
      // My dialog initiates and waits for the next message from the user. 
      await context.PostAsync("Hi I am Movie Booking Bot created by Neel.");
      await Respond(context);
      // When a message arrives, call MessageReceivedAsync. 
      context.Wait(MessageReceivedAsync);
    }

   private static async Task Respond(IDialogContext context)
   {
     var userName = string.Empty;
     context.UserData.TryGetValue<string>("Name", out userName);
     if (string.IsNullOrEmpty(userName))
     {
       await context.PostAsync("What is your Name?");
       context.UserData.SetValue<bool>("GetName", true);
     }
     else
     {
       await context.PostAsync(string.Format("Hi {0}.", userName));
     }
  }

  public virtual async Task MessageReceivedAsync(IDialogContext context, IAwaitable<IMessageActivity> result)
  {
    var message = await result;// We've got a message! 
    var userName = String.Empty;
    var getName = false;
    context.UserData.TryGetValue<string>("Name", out userName);
    context.UserData.TryGetValue<bool>("GetName", out getName);
    if (getName)
    {
      userName = message.Text;
      context.UserData.SetValue<string>("Name", userName);
      context.UserData.SetValue<bool>("GetName", false);
    }
    await Respond(context);
    context.Done(message);
  }
 }
}

Once this is done, let us modify MessagesController class which is the heart of the application and our above classes will be called within this controller class.

Write code to MessagesController.cs class:

using NeelBotDemo.Dialogs;
using NeelBotDemo.FormFlow;
using Microsoft.Bot.Builder.Dialogs;
using Microsoft.Bot.Builder.FormFlow;
using Microsoft.Bot.Connector;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using System.Web.Http;

namespace NeelBotDemo
{
 [BotAuthentication]
 public class MessagesController : ApiController
 {
   /// <summary> 
   /// POST: api/Messages 
   /// Receive a message from a user and reply to it 
   /// </summary> 
   public async Task<HttpResponseMessage> Post([FromBody]Activity activity)
   {
     if (activity.Type == ActivityTypes.Message)
     {
       await Conversation.SendAsync(activity, () => MovieBotDialog.dialog);
     }
    else
    {
      HandleSystemMessage(activity);
    }
   var response = Request.CreateResponse(HttpStatusCode.OK);
   return response;
 }
internal static IDialog<MovieBooking> MakeRootDialog()
{
   return Chain.From(() => FormDialog.FromForm(MovieBooking.BuildForm));
}
private Activity HandleSystemMessage(Activity message)
{
  if (message.Type == ActivityTypes.DeleteUserData)
  {
    // Implement user deletion here 
    // If we handle user deletion, return a real message 
  }
  else if (message.Type == ActivityTypes.ConversationUpdate)
  {
    // Handle conversation state changes, like members being added and removed 
    // Use Activity.MembersAdded and Activity.MembersRemoved and Activity.Action for info 
    // Not available in all channels 
  }
  else if (message.Type == ActivityTypes.ContactRelationUpdate)
  {
    // Handle add/remove from contact lists 
    // Activity.From + Activity.Action represent what happened 
  }
  else if (message.Type == ActivityTypes.Typing)
  {
    // Handle knowing tha the user is typing 
  }
  else if (message.Type == ActivityTypes.Ping)
  {
  }

 return null;
 }
}
}

Whatever response User sends, it will first come to Post method of MessagesController class and then it will invoke MovieBotDialog class.

Our Bot is ready now, just click on start button and you will see below window:

bot9

As I explained in my previous post, we will require a Bot Emulator to test our bot.

So we will start testing our bot by providing our bot URL which is http://localhost:3979/api/messages

Let us test our Bot:

We will start by:

  • Writing hi
  • Bot will greet and will ask your name
  • Once we provide Name, Bot will store our name in user context and will tell us to continue
  • Once we type anything, it will ask to select the city:

flow3

Continue clicking on your selection, a different selection box(forms) will appear which we have given in the class MovieBooking.

Screenshots of Bot for the whole conversation:

flow4

flow5

flow6

flow7

As you can see in above image, whole summery of your inputs have been given, this is awesome.

Say Yes and your booking is successful:

flow8

As you can see the name Neel in the last sentence, it was collected from the user data context.

Congratulations, you have just created a Bot with FromFlow.

Please contact me or write the comment if you have doubt for any of the line of code above.

NOTE – the Whole code is available here.

Hope it helps.

 

 

Naive Bayes in sentiment analysis: Machine Learning Questions & Answers Part – IV

naive6

This is the fourth post on Machine Learning Questions and Answers series. Look here for previous posts.

In this post, we will see some questions related to Naive Bayes Algorithm.

  1. What is Naive Bayes Algorithm?

Answer –

One liner for Naive Bayes Algorithm:

Assuming all the feature are independent and are equally important and predicting the things based on prior knowledge and these independence assumptions.

In other words:

  • The Naive Bayes Algorithm is a Machine Learning Algorithm for Classification problems
  • It is primarily used for text classification
  • A few examples are Sentimental analysis, spam filtration, classifying news articles etc
  • In short, it is a probabilistic classifier

For example:

  • You went to a restaurant for dinner
  • You were told to give a feedback
  • You are quite happy with the experience and gives feedback as “Happy with the food”
  • As per Naive Bayes, all the words given in the feedback are independent of each other and equally important
  • So a Naive Bayes Algorithm can be applied here which will then be matched with some historical data of those words
  • And at the end of the calculation, we can predict whether the feedback given is positive or negative
  • You can get more details in the same post below

2) How could sentimental analysis be used using Naive Bayes Algorithm?

Answer – 

  • For example, our job is to take a comment and classify whether it is a positive or negative comment
  • So we have “comment” as problem instance and Positive or negative as labels
  • As I have explained in my previous posts, to solve above classification problem, we will require a training data which has a large body of texts which are labeled as positive or negative
  • For example word, “Good” would be labeled as Positive and word “Bad” would be labeled as Negative

Next step would be to fit our data to a standard Machine Learning Algorithm, we will use Naive Bayes Algorithm here.

  • In the training phase, the algorithm tries to collect the information to classify any new text as positive or negative
  • Let us assume that we have completed the training phase and it resulted in 55% positive words and 45% negative words
  • So when a new comment comes, we can divide that comment into different words and we can compute a positive score and a negative score for every word

Let us see how a comment can be recognized whether it is a positive comment or a negative comment.

We have below result from our training phase:

naive1

Now a new comment comes and we need to identify whether it is positive or negative.

The comment is “Happy with the Food”

  • Naive Bayes Algorithm will calculate a positivity score and Negativity score
  • It does this using the pieces of information we got from training phase

Let us first calculate the Positivity score for above comment:

  • We will divide the comment into mainly 2 major words, “Happy” and “Food”
  • We can ignore some specific words like “with” “the” etc
  • Positive score can be computed by multiplying Positive score of “Happy” with positive score of “Food” along with the overall probability that the comment is positive

For example:

Below image would explain how to calculate the positive score for the comment “Happy with the Food”:

naive2

As you can see after the multiplication, positive score came as 0.24

Similarly, we can calculate the negative score for the same comment.

Basically Negative score for a word from the training model can be calculated by doing minus from the positive score as explained below:

naive3

So if we try to calculate negative score as we did above then:

naive4

As you can see we got Negative score as 0.02

Here we have taken the example of only 2 words, but it can be computed with n number of words.

Final score is, the comment “Happy with the Food” is positive because positive score 0.24 is greater than Negative score 0.02

naive5

This is how Naive Bayes works with Sentimental Analysis.

3) Why Naive Bayes is called naive?

Answer – 

The Naive Bayes is called Naive because:

  • It makes the assumptions that occurrence of certain features is independent of the occurrence of other features

For example:

  • Your job is to identify the fruit based on its color, shape and taste
  • Then Orange colored, spherical and tangy fruit is most likely be an Orange fruit
  • Even if these features depend on each other, or in the presence of the other features, all these properties individually contribute to the probability that this fruit is Orange
  • For example, if there is a fruit which is orange in shape but it is not spherical in shape and it does not taste tangy but still it could be an Orange fruit

Hope it helps.

 

 

 

 

 

 

 

 

Solution for the error “No executable found matching command “dotnet-ef” : .Net Core Errors Part I

ASP.NET-Core-Logo-blue

.Net Core was introduced in last few months and people has started adopting it.

In this series of post, I will put some frequent issues during .Net Core development and some important topics for the Core.

In this post we will see how to resolve the error:

No executable found matching command “dotnet-ef”

This error comes when we want to migrate the database with the models and when we use below command:

dotnet ef migrations add FirstMigration

Reason:

Because dotnet-ef tool might have not been added along with your template

This error comes if you have not added Microsoft.EntityFrameworkCore.Tools in your Tools section of project.json file or if you have not added DotNetCliToolReference into the ItemGroup section, I will explain more in below solution.

Solution:

Below are some steps which you can try to resolve the error:

For .Net Core 2.0

  1. Add Microsoft.EntityFrameworkCore.SqlServer in your project if it is not added yet, you can add them either using Nuget package by running below command:
dotnet add package Microsoft.EntityFrameworkCore.SqlServer

2. Add DotNetCliToolReference into the ItemGroup as below in your csproj file( for 99% cases, this should work):

<ItemGroup>
  <DotNetCliToolReference 
      Include="Microsoft.EntityFrameworkCore.Tools" 
      Version="2.0.0" />
  <DotNetCliToolReference 
      Include="Microsoft.EntityFrameworkCore.Tools.DotNet" 
      Version="2.0.0" /> 
</ ItemGroup>

3. If it does not work still then add Microsoft.EntityFrameworkCore.Design Nuget package:

dotnet add package Microsoft.EntityFrameworkCore.Design

For .Net Core 1.0:

  1.  Add Microsoft.EntityFrameworkCore.Tools and Microsoft.EntityFrameworkCore.Tools.DotNet in your project.json file, you can add them either using Nuget package or manually as below:
"tools": {
   "Microsoft.EntityFrameworkCore.Tools": "1.0.0"
   "Microsoft.EntityFrameworkCore.Tools.DotNet": "1.0.0"
 }

2. Add DotNetCliToolReference into the ItemGroup as below in your csproj file( for 99% cases, this should work):

<ItemGroup>
    <DotNetCliToolReference Include="Microsoft.EntityFrameworkCore.Tools.DotNet">
      <Version>1.0.0-*</Version>
</DotNetCliToolReference>
    </ ItemGroup>

Important Notes –

  • Make sure that you are running the command from the folder that contains the project where the Tools package is referenced

The error should be resolved now.

 

Create your first Bot using Visual Studio 2017: Step by step guide

bot17

By looking at how fast the companies adopting the Bots, it is really the best time for you to start learning Bot framework and start adopting Bots for your business.

Some pain in the real world without the Bots:

  • You have to read the whole FAQ to find some specific information for any website or any company
  • You have to wait for next business day to start to get answers to your queries

bot15

  • You have to send emails to get some information to send some information
  • You have to do manual work to answer some repetitive questions
  • More manpower would require if the number of questions increases suddenly:

bot16

This would eventually affect your business. Time to try the Bots.

Let us see what the Bots are:

An Internet bot, also known as web robotWWW robot or simply bot, is a software application that runs automated tasks (scripts) over the Internet. Typically, bots perform tasks that are both simple and structurally repetitive, at a much higher rate than would be possible for a human alone.

In simple words, Bots are something that can be integrated with your website and they can answer the questions posted by the users without the need of the humans

Let us see how to create simple Bot application using Visual Studio 2017.

prerequisite:

Also if you want to have Bot Application as a template then as a workaround just download this  (download would start once you click on the link) project and put the extracted folder into below location:

C:\Users\YourName\Documents\Visual Studio 2015\Templates\ProjectTemplates\Visual C#

Once this is done, you can see Bot Application template as shown below:

bot2

Click on Bot Application and then it will create a sample project which has the structure as below:

bot3

Here MessagesController is created by default and it is the main entry point of the application.

If you open MessagesController, it will look blood red because it cannot find missing Nuget packages:

bot4

You just need to restore those missing Nuget packages by opening Nuget Manager:

bot5

You may encounter an error which would be as below:

CS0117 ‘Task’ does not contain a definition for ‘CompletedTask’ NeelBotDemo c:\users\NeelB\documents\visual studio 2017\Projects\NeelBotDemo\NeelBotDemo\Dialogs\RootDialog.cs 15 Active

This error comes because Task.CompletedTask is a static property added in .NET 4.6 and your application may have .NET 4.5

You need to change your application’s Target framework and make it 4.6.1 as below:

bot6

Your solution should build properly now.

Now we will make changes in the default code and will modify as per our need.

Open RootDialog.cs class which is in the Dialogs folder.

Replace the code of method with below code:

private async Task MessageReceivedAsync(IDialogContext context, IAwaitable<object> result)
{
  var activity = await result as Activity;
  // calculate something for us to return 
  int length = (activity.Text ?? string.Empty).Length;
  // return our reply to the user 
  //test 
  if (activity.Text.Contains("morning"))
  {
     await context.PostAsync("Good Morning , Have a nice Day");
  }
  //test 
  else if (activity.Text.Contains("night"))
  {
     await context.PostAsync("Good night and Sweetest Dreams");
  }
  else if (activity.Text.Contains("who are you"))
  {
     await context.PostAsync("I am a Bot created by Neel");
  }
  else if (activity.Text.Contains("date"))
  {
     await context.PostAsync(DateTime.Now.ToString());
  }
  else
  {
     await context.PostAsync($"You sent {activity.Text} which was {length} characters");
  }
context.Wait(MessageReceivedAsync);
}

Here we are telling Bot, what it should answer when there are some specific keywords are there in the message.

Once you made the changes, just run your Bot application. It will have landing page as below:

bot9

At this point, your bot is ready to be used. We need an emulator to test our bot.

If we want to test our bots locally then Bot emulator is the best option.

The Bot Framework Emulator is a desktop application that allows bot developers to test and debug their bots on localhost or running remotely through a tunnel.

As we mentioned on top of the post, you can download the Bot emulator from here. Or you can click below, it will start the download automatically:

botframework-emulator-Setup-3.5.33.exe

Click on exe, it will start the installation:

bot10

And once the installation is done, it will have a landing page as below, here you need to give the URL of your bot application(http://localhost:3979/api/messages):

bot13

It will ask you for Microsoft App Id and password, but for now, do not give anything there and click on CONNECT.

Now your bot is ready to be tested.

Give something that you have added to your code and the bot will respond as per your input in code:

bot14

As you can see it answered all of those which we have given in above code and for rest of the things, it will answer as: “You sent {input} which was {length} character”

Congratulations, you just created your first Bot 🙂

You can integrate Microsoft Cognitive APIs into your Bot application, I have written a post on the same which you can find here.

In my upcoming posts, I will share my experiments with the Bots.

Important Note – Microsoft team has created Bot Builder SDK for .NET so that it is easy for us to develop the bots. But if you want to know what is happening behind the curtain then you can have a look here: https://github.com/Microsoft/BotBuilder/tree/65d4e985b68e4fce5a5e0ac81de198c0a95ac7bb/CSharp/Library, all the libraries are here.

For example, you may have seen I used BotAuthentication attribute above the action, you can find the code for the same attribute here: https://github.com/Microsoft/BotBuilder/blob/65d4e985b68e4fce5a5e0ac81de198c0a95ac7bb/CSharp/Library/Microsoft.Bot.Connector.AspNetCore/BotAuthentication.cs

Hope it helps.

 

How to Choose ML Algorithm: Machine Learning Questions & Answers Part – III

ML-06

This is the third post on Machine Learning Questions and Answers series. Look here for previous posts.

The fundamental requirement to start any Machine Learning project is to identify which algorithm we should apply for the business problem on which we are starting the Machine Learning project.

For that, you must have the ability to pick the algorithm by looking at the problem. You should have that vision to decide whether we should choose Classification or Regression or Clustering or Recordation etc.

Today’s question is somewhat around that.

  1. How to choose Machine Learning Algorithm from the problem given by Business?

Answer – 

The Very first step would be to create a Problem Statement from the problem given by the Business.

Importance of the Problem Statement

  • Whenever you start working on any Machine Learning project, you first need to understand what the problem is
  • You should be able to understand what type of problem we want to solve and it mainly depends on how you set the problem statement
  • Setting a problem statement includes the input and output of the business problem you want to solve
  • Problem statement must be concrete and it should not have multiple sentences in it
  • For example, if your company is getting many harmful spams or some fake emails which may harm your organization, you require an algorithm which can identify what are the spam\fake mails out of all the emails. So, in this case, an email would be the input and identification whether the email is spam\fake is the output. So the Problem statement would be “Is the mail spam\fake or not?

Once the problem statement is defined, you should be able to identify which algorithm you should choose among below types:

  • Any Classification Algorithm
  • Any Clustering Algorithm
  • Any Regression Algorithm
  • Any Recommendation Algorithm

When to use Classification Algorithms:

For example, your problem statement is a question to classify something(ie Is this good or bad?) or when your goal is to predict discrete values, e.g. {1,0}, {True, False}, {Spam, No Spam}

Classification is a Supervised Learning.

classification

Real Time Examples:

  • Is this mail a spam or not?
  • Is the customer happy or not?
  • After entering out of a restaurant, was the person looking satisfied or not? Here Resturant can be replaced with anything like Bank, shop, mail etc
  • Is the feedback provided by the customer positive or negative?
  • If the trading day is an Up-day or a down-day?
  • Classification algorithms are used by banks to classify loan applicants by their probability of defaulting payments

All above problem statement has some certain pattern. They involve taking an object or entity and classifying it. For example in Spam detection, we are classifying an Email

And on the other hand, you have something which needs to be classified. For example, if we are doing Binary classification then there might be 2 categories like the tweet is positive or not or if we are doing the classification with multiple categories then it needs to classify accordingly.

Example of Classification Algorithm:

  • K – Nearest neighbor
  • Decision Trees
  • Bayesian Classifier

When to use Clustering Algorithms:

For example, there is a large group of users and you want to divide them into particular groups based on some common attributes. But the key part is when the groups to be divided into are unknown beforehand. So you should go with Clustering Algorithm when there are not known attributes beforehand.

If there are questions like how is this organized or grouping something or concentrating on particular groups etc in your problem statement then you should go with Clustering.

Classification is an UnSupervised Learning.

ml35

Real Time Examples:

  • When you have large social network site and you want to divide the users on basis of the Likes they made on the post or on basis of Demographics, so it helps to identify the meaningful groups
  • A company got to know particular products are not making sales as expected so those products can be clustered.  then only the cluster of such products would have to be checked rather than comparing the sales value of all the products
  • Most of the search engines like Yahoo, Google uses Clustering Algorithms to cluster web pages by similarity and identify the ‘relevance rate’ of search results. This helps search engines reduce the computational time for the users.
  • We all know Dominos guarantees you to deliver pizza in 30 minutes, they use Clustering to identify the Pizza shop location such a way that traveling time of the Pizza guy is minimized. That is the reason there are so sure that they will reach customers within 30 minutes

You should go with the Clustering when the major focus is to create groups which have similar attributes. For example, you’re given a set of history transactions which recorded who bought what. By using clustering techniques, you can tell the segmentation of your customers.

Example of Clustering Algorithm:

  • K- Means Algorithm
  • Expectation maximization

When to use Regression Algorithms:

For example, you want to compute some continuous value as compared to Classification where the output is categoric. So whenever you are told to predict some future value of a process which is currently running, you can go with Regression Algorithm.

Here you deal with numbers and there might be questions about How much or how many or how long or Impact of something on something else in your problem statement, in such cases you should go with Regression.

Classification is a Supervised Learning.

ml36

Real Time Examples:

  • What will be the value of Doller equivalent to the Bitcoin on any particular future date
  • How long it would take me to go Home from my office
  • What would be the sales of particular product next month
  • Impact of blood alcohol content on coordination
  • A credit card company applied regression analysis to predict monthly gift card sales and improve yearly revenue projections

As you can see in above all examples, we have some continuous process like BitCoin price, Compute Time, Sales etc and the output depends upon some certain inputs. For example in case of Compute time, it depends on the time of the day when I want to travel + the distance + weather etc.

Example of Regression Algorithm:

  • Linear Regression Algorithm
  • Logistic Regression Algorithm
  • Polynomial Regression Algorithm etc

When to use Recommendation Algorithms:

For example, when you want to determine what kind of theme a user would like in future based on the user’s past behavior. Like as Collaborative filtering(for example, User like you also liked kind of things)

So when there are some questions like what the user should do next or suggest something to the user or top choices of particular users etc in your problem statement then you should go with Recommendation.

Classification is an UnSupervised Learning.

ml37

Real Time Examples:

  • If a user buys the Washing Machine, what else he would buy in future
  • What are top 10 choices of books for a particular user
  • What kind of artist the user would like when he comes back on any music related applications
  • Providing user A the list of item which has been bought by another user B whose behavior is almost similar to user A

As you can see above examples, it mainly based on any particular user and his\her historical behavior.

Example of Recommendation Algorithm:

  • Collaborative filtering Algorithm
  • Logistic Regression Algorithm
  • Polynomial Regression Algorithm etc

Conclusion:

Whenever you want to identify which Algorithm to use:

  • Set one concrete Problem statement out of the problem given to you
  • The choice you made here will completely determine what would be the next steps
  • Make the choice which Algorithm you should choice by looking at the problem statement like what exactly is needed
  • Choose the type(ie Clustering) and choose the Algorithm(ie K-Means Algorithm) and start your project

Hope it helps.

 

 

 

 

Overfitting in Machine Learning: Machine Learning Questions & Answers Part – II

over9

I have started the series of Machine Learning Questions and Answers, you can find the first post here.

Let us see some more questions and answers.

  1. What is OverFitting in Machine Learning? OR Is OverFitting good or bad in Machine Learning?

Answer – 

Overfitting is not good for Machine learning projects.

As the name suggests, it is nothing but trying to fit something over than required.

One liner for Overfitting:

When you remembered but have not learned something, thus you are not prepared for the future

Definition:

  • So Overfitting occurs When we capture low-level details in a particular data set but we fail to capture the higher level, more abstract details of that data set. Thus it creates the problem for the future examples
  • In other words, Overfitting happens when a model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data
  • And as per Wikipedia –  Overfitting is the production of an analysis that corresponds too closely or exactly to a particular set of data, and may, therefore, fail to fit additional data or predict future observations reliably.

Real life example of Overfitting:

  1. You started reading a chemistry book
  2. You read every single low-level detail like every single letter and digit
  3. You read the whole book from start to end
  4. But what if you can not think about the bigger picture like relating different topics with each other, how one topic is related to another topic etc
  5. Thus you have remembered the book but have not learned it
  6. So if someone comes and asks you something from that book, it is not sure whether you can answer all of them

Let us see some examples:

You have some data and you put it into the X and Y axis as below:

over2

If you try to separate the X and 0 something like below:

over3

Though it looks valid, it looks so bad and may create a problem in future. This may happen when you take data too literally. So in above example, this may work perfectly for current data but what if some new data is added, it will not work perfectly.

When you can just take something simple as below:

over4

By this, if you add some more data then also the distance of 0 would be lesser compared to the line which we drew above.

In short, prepare for future instead of giving too much time to the present.

Let us take some visual examples:

For example, you want to create a program which can identify the lion.

In this case, we are overfitting when we collect very basic and complex data like height and width of the lion some other very minor and deep details as below:

over5

So the program will perfectly identify a lion which comes with little bit similar details as mentioned above, but this would become useless incase of identifying some new examples like a White Lion.

over6

The program would fail to identify White Lion because it was trained with some unnecessary deep details instead of some abstract and useful details.

2) How can we avoid Overfitting?

Answer – 

We can avoid Overfitting by splitting the data 3 way:

over7

  • Training set
  • Cross-Validation
  • Test set

This way we are assuring that the model is not dependant on any particular set

Apart from this, you can:

  • Keep it Simple
  • Feature selection: consider using fewer feature combinations and decrease the number of numeric attributes bins
  • Increase the amount of regularization used
  • Increase the amount of training data examples
  • Increase the number of passes on the existing training data

Hope it helps

 

Classification and Clustering : Machine Learning Questions & Answers Part – I

int3

I have recently started posting related to Machine Learning and I got some very positive feedback from people because they are liking the way I explain Machine Learning related topics in simple words.

As per the demand, I am starting the series of Machine Learning Questions and Answers.

I will keep on posting the questions along with the answers here as soon as I get to know it.

So let us start:

1) What are the types of Machine Learning and what is the difference between Supervised Machine Learning and UnSupervised Machine learning?

Answer – For this, I have already written a post which you can find here.

2) What is the difference between Classification and Clustering?

Answer –

One liner for Classification:

Classifying data into pre-defined categories

One liner for Clustering:

Grouping data into a set of categories

Key difference:

Classification is taking data and putting it into pre-defined categories and in Clustering the set of categories, that you want to group the data into, is not known beforehand.

Let us go a bit deeper into Classification first:

classification

  • In classification, you would start with one instance(one object) to be classified
  • You would classify it into pre-defined categories which are nothing but the labels
  • Do this based on the training data which has already been classified
  • For Example – In sentiment analysis, you would classify one comment as positive or negative and you would do this based on the set of training data which are already been classified into positive and negative comments
  • So if you understood Supervised Machine Learning then you would realize that Classification is nothing but the Supervised Machine Learning

Simple understanding with an example:

You give your algorithm(your friend) some data(Set of People), called as Training data, and made him learn which data corresponds to which label(Male or Female). Then you point your algorithm to certain data, called as Test data, and ask it to determine whether it is Male or Female. The better your teaching is, the better it’s prediction.

Some real-life examples:

  • If the e-mail is a spam or not
  • Is the comment on a Facebook post or a Tweet on Twitter is positive or negative
  • If the trading day is an Up-day or a down-day
  • Handwritten Digit Recognition
  • Speech Recognition
  • Image Recognition

Example of Classification Algorithm:

  • K – Nearest neighbor
  • Decision Trees
  • Bayesian Classifier

Steps of Classification Setup:

INT1

  1. The problem has to be defined first
  2. Then you would represent your data in the form of Numerical attributes called Features. This is done both for the training data which has already been classified and the test data which has to be classified in the future
  3. You would take your training data and feed it into a classification algorithm to train a model
  4. Take new instance that needs to be classified or take the test data and pass it to classifier to classify

Now let us see something more about Clustering:

clustering

  • Instead of taking single Instance(As the case of Classification above), we are taking large number of instances
  • We divide these number of instances into the groups
  • So as we had pre-defined categories in Classification, in clustering the groups are unknown beforehand
  • Basically, we do not know before the clusters are formed, what to call those clusters because we would not know until then, what would be the common categories inside these clusters
  • Yes, it is UnSupervised Machine Learning

Simple understanding with an example:

In Clustering, you provide the data(Set of people) to the algorithm(your friend) and ask it to group the data.

Now, it’s up to algorithm to decide what’s the best way to the group is? (Gender, Color or age group).

Again, you can definitely influence the decision made by the algorithm by providing extra inputs.

Some Real-life examples:

  • How we can divide set of articles such that those articles have the same theme(we do not know the theme of the articles ahead of time)
  • Identifying groups of houses according to their house type, value and geographical location
  • Earthquake epicenters to identify dangerous zones
  • Putting telephone towers in a new city using clustering such that all its user receives optimum single strength

Example of Clustering Algorithm:

  • K- Means Algorithm
  • Expectation maximization

Steps of Clustering Setup:

int2

  1. You would start with the problem statement, which is the database which needs to be clustered
  2. Then you would represent points in that dataset using features.
  3. No training step here
  4. You would directly feed the data into Clustering algorithm to find the final clusters, without any training steps

3) Can Classifier and Clustering go hand an hand OR Can Classifier and Clustering work together?

Answer – Yes they can.

For example, you have set of articles -> you divide these articles into the clusters based on the tags  -> The Articles are grouped based on the tags

Now you have an article -> Article is sent to Classifier and Classifier will assign one of the tags from the tags that are discovered during Clustering above – > Tag is identified

So basically, the articles which are grouped based on the tags into different clusters are becoming the training data for the Classifier.

Conclusion:

  • Classification assigns the category to 1 new item, based on already labeled items while Clustering takes a bunch of unlabeled items and divide them into the categories
  • In Classification, the categories\groups to be divided are known beforehand while in Clustering, the categories\groups to be divided are unknown beforehand
  • In Classification, there are 2 phases – Training phase and then the test phase while in Clustering, there is only 1 phase – dividing of training data in clusters
  • Classification is Supervised Learning while Clustering is UnSupervised Learning.

Hope it helps.

 

 

 

Linear Regression in simple words: Machine Learning Algorithm part I

ml31

I have written a post in which I explained Machine Learning in simple words. You can find the post here.

Heart of the Machine Learning is the bunch of Algorithms. Algorithm plays very important role in creating the models. Nowadays different languages like Python, R and different tools like Azure, AWS has made our life so easy if we want to create Machine Learning projects.

But one should understand the algorithm instead of just using pre-existing libraries which above languages have already created.

In this article, I will try to explain Linear Regression.

Let us go back to our school days. You might remember syntax:

Y = MX + B

where:

  • B is the intercept,
  • M is the Slope, can be positive or negative
  • X is Independent variable
  • Y is dependent variable

So if you have X, you can figure out what Y is.

In simple words “Linear Regression” is a method to predict dependent variable (Y) based on values of independent variables (X). 

Situation 1:

If X increases and Y also increases, it is called positive relation:

ml22

Situation B:

If X increases but Y decreases, it is called negative relation:

ml23

Let us see how to create a regression line:

To conduct regression, we require different observation. We can put those observations between X and Y:

ml24

Once all the observations are placed correctly, we can create the line which will fit all those observation dots and this is called the Regression line:

ml25

As we know, all the observation would never be in the straight line, there is always a difference between estimated value and an actual value. In the end, we are required to minimize the difference between estimated value and an actual value. We will call this difference as errors:

ML26

We would target to minimize these errors and above line has many errors when we compare actual with estimated values:

ml27

Let us take some examples to understand the Positive relationship and Negative relationship.

For example, if we study more our grades would increase:

ml28

It is a positive relationship where:

  • y is estimated grades
  • x is study time
  • b0, we can derive mathematically and it is the y interceptor
  • b1, this can also be derived mathematically, it is the slope

In simple words, if your study time is 0, grades are 10% and if you increase study time by 10 hours, grades would be greater than 10% and grades can be counted using above syntax.

Though there might be lot other features which affect the grades, for simplicity, we have only considered study time as the only feature. This is also called Univariate Linear Regression.

Multiple Linear Regression

We can increase our feature set by selecting more parameters like IQ of the person, the interest in the subject etc. For example, we can plot the grades against the interest of the person in particular subject and study time on a single graph, where the vertical axis plots grades and the two horizontal axes the interest in the subject and study time:

In this case, we can again fit a predictor to the data. But instead of drawing a line through the data we have to draw a plane through the data because the function that best predicts the grades is a function of two variables.

Now, let us see the Negative relationship:

If you spend more time on Facebook, grades would decrease:

ml29

As you can see above, if x increases, it would be multiplied by -b1(slope). Thus y decreases.

X is a dependent variable which we can manipulate, control, change and Y is a dependent variable which is nothing but the outcome of X’s activity.

Let us see some calculation of Study example:

ml30

Hope it helps.

 

Create a free website or blog at WordPress.com.

Up ↑