How can you predict any algorithm that how it works?

I am seeing this I’m seeing this distribution this way I am seeing this kind of overlap and redo this I’m seeing this kind of overlap I am when I use another dimension in the other dimension also I am seeing an overlap but when I put them together in that mathematical space the two Gaussian become linearly separable because of that linear separability now possible amongst the classes on the dimensions look at this dimension this one.

I don’t know what which one is this one is not going to help you separate the orange from the blue doesn’t matter as long as I have this dimension with me which separates orange from the Blues, so one dimension will compensate for the other dimension so overall these weak predictors put together to become strong predictors exactly overall these weak predictors put together to become strong predictors and as you will notice.

When it goes down the line look at let’s look at the data types they we don’t have any nonnumerical columns here everything is numerical keep in mind all algorithms need numerical columns only numerical columns could be categorical or real numbers come down so I am separating the X variables independent variables the target is the cultivator I am splitting data into training set tested I was splitting my data into training septa set you’re of all of you with me on this okay you can actually download the data set from UCI if you wish to this data set you can download from UCI and run the code very good.

I instantiate the Gaussian noise base here there are different variants of Nye base this nine base assumes that the dimensions are Gaussian on each one of these the distributions are Gaussian on each one of the dimension and we meet that requirement very well on all the dimensions the classes are distributed in almost Gaussian ways Gaussian means looks like a normal distribution.

So perfect case for Gaussian I base so I am calling Gaussian evasive there are other variants like multinomial naive Bayes and other my base which you can make use of so I am instantiating the model Here I am going to do the model fit here this is where our law this my base likelihood ratios will be calculated and then I look at this on the training set itself I am doing testing it’s giving me 97% accuracy why am I doing this I’ll tell you in a minute while I can also do this model on the test data.

Here I am running the model on the test data okay and in the matrix, there is a function called classification report which gives me all this matrix recall precision everything in one shot if you are interested you can separately print the matrix also confusion matrix now look at this confusion matrix look at this look at the class level recall any problems he knew it holy any problems from SQL and import Gaussian okay all right now look at this there are three classes here one two three and look at the class level recall what is class level recall accuracy at class levels almost 100% one means hundred ninety five percent hundred percent.

Each one of the classes it’s able to accurately classify with a degree of this is what we wanted in Pisa diabetes it’s giving us a corrosion vine dataset okay I’ll talk about this micro average and all these things down the line we’ll talk about this later right now I just look at this recall matrix I have not told you what precision is and look at the confusion matrix an actual number of the class an in your test record is 23 all of them have been classified as a class.

An actual number of Records for Class B in your test data is 19 it has correctly classified 1812 it has correctly classified 12 you can achieve this kind of score in this model even though the dimensions are overlapping the classes are overlapping and all the dimensions because of some dimensions which come together to make strong predictors so you should understand the difference between Lima disabilities and this in both the cases the classes are overlapping, but there’s one very important critical difference and it’s that difference which is making this right.

You can run the same algorithm base on Pisa diabetes and see what is the score you get there your job here is to select the right attributes that this is you will never get data on a plate when you are doing real-life products this is one big classroom but when you do sit down and do real-life projects you customer sees like to suppose a simple example can you use data science to improve my customer cell CSAT ratings from the current three point five to four point five out of five can you do something using data science to help me improve my customer rating in our technical support these are the kind of requirements which will come to your data will never come to you.

Now your job is to see site rating customer rating for technical support what kind of data you need not desired you have to first guess what kind of data I need your domain expertise will help you if you don’t have it you need to have domain experts so for season rating in customer tech support what kind of data elite can somebody give go back to the past decades okay go back to the past ticket what kind of data you need to go back to the past decades from the past decades you collect what kind of ticket p1 p2 p3 tickets.

I need to find ticket classes then whether the ticket belongs to hardware-software or something else I need those things and various other you will decide what kind of data you need the next challenge will be very will you get the data from some data will be available within the organization some data will be available outside the organization some data will be available with the customer so getting this stakeholder to give the data to us will be such a challenge so all your soft skills will come into play right.

So once the data comes in you have to first establish the reliability of the data what if the tech support department has given you data where the customer is very happy they are not shared with you the dirty data your model will go for it to us those are the challenges which you will face as a data scientist once the data comes to you on this way attributes that you have you will do this analysis using pair plot and other techniques is this column good or that column good which column should I use to define customer satisfaction.

I come from the IT world just like you do many of you do we people have done a lot of projects in Java and C in all these things there the project starts with customer requirements which would turn into technical requirements that we turn into design requirements that turn into code specs coding requirements that turn into unit testing integration testing and finally sub Murcia then we go for acceptance testing in acceptance testing usually it bombs right.

The reason why it bombs is we look at data only in the last stage may we come to acceptance testing we ask customers to give us some data in data science projects your project will start with data the project will end with the data is the core you will be revolving around the data sets say 80% of your effort estimated effort in data science project you will see that when you do the capstone project will go in getting the data.

How to collect information through algorithms?

When you go to the web every one of you gets personalized ads based on what kind of size had you visited before right all of that information is collected algorithms are run on that machine learning algorithms are all of that and then a very specific for a segment very specific types of ads are wrong so I mean.

Now as we go along it’s not a choice for any business to not use machine learning they have to retain the competitive edge everyone is running these machine learning algorithms right but the thing with machine learning is that and in business scenarios in most business scenarios an accuracy or 60 or 70% is acceptable right so if you want to target some population for your marketing and decide which marketing campaign to run even.

If you’re going off by 20-30 percent that is fine right as long as what you’re most of your population is the target audience, so that’s the reason machine learning techniques are now widely being used in all business decisions where 60 to 70 percent accuracy is acceptable right that actually if it goes beyond 70 beyond let’s say 80 85 percent people often worry about that the algorithm is overfitting right then all of these concepts are you now covered in the course and as you learn about machine learning you learn about these concepts right, but that’s how it is that 60 70 percent accuracy is when machine learning you got them to see used now what is deep learning.

Now when you want this accuracy to go higher from 60 70 percent – let’s say 99 percent plus right and that’s a big jump the kind of techniques that you start using is very very different and all of those techniques are grouped under the term deep learning so I described to you the example of Amazon Go stores right there you cannot live with a 70% accuracy you cannot be billing seven customers right and three customers strongly right because that’s the perception that will build around wrong billing will build up and then your model will never succeed or if you’re doing image recognition you cannot be okay with seven people being identified correctly and three people not being.

If I correctly write so in some of these Vista scenarios and some of these use cases you need very very high accuracy upwards of ninety-nine percent and that’s where you start using the deep structured learning algorithms and this whole broad family of techniques under deep running.

So that’s the difference so to summarize I mean that was the difference between her xi artificial intelligence machine learning and deep learning artificial intelligence being the broad umbrella for any machine to do a specific task under a specific intake machine learning so slightly more advanced techniques which gave about 60 70 80 percent accuracy and from there if you want to go to 99 percent accuracy in above then you have to start using deep learning techniques which are again you have heard about it neural networks convoluted neural networks all of that right.

So some of these from the techniques using deep learning and that’s the V sticking on Alfred deep learning okay so let’s so this it was the session was the previous section was more about understanding what is AI right and just getting a little more sense on what is being talked about and what is a distinction between some of these terms right in this section we’ll talk about how does it apply to us how does it apply to the companies how does it change how the overall environment around us is changing right and how does it translate into jobs and opportunities right so in this slide what we see is a path weight from business intelligence and this is the path which every company will have to follow right in the whole journey towards AI.

So back in the 90s most companies were using data and visualizing the data and most of these companies were doing it not because they wanted to do it but because there was a regulation there was if they were required to do it by regulation right so, for example, let’s say putting the data together for financial statements banks would you know reporting there are risk metrics to the regulator’s right former companies were reporting some of the test metrics right.

So they were just gathering data and the max they where Boeing was just doing a visualization of DE tightness and then like obviously all of us have seen the trend charts and pie charts just very very basic visualization to understand what is happening right just figuring out sales forecast is increasing so one of that is clubbed under business intelligence right and they got more and more sophisticated they started using some of these security tools to do it in a better way but the fullest hence was to just look at the data and understand there any patterns in the data.
That used to be in the 90s for most organizations but as they moved along the second stage was using the data to do predictions, so people started building very very simple kind of models with this data or you know just you know looking at a trend in predicting it tears on the line.

All of these small prediction modelings started happening which also at that point of time involved let’s say regression allows the integration of all of that so so all of that started happening in the early 2000s which are found under Dixon breakage modeling right and then after that as the cost of storing data declined rapidly and also the cost of computation became much much you know cheaper and as we saw that most organizations were starting to the IT systems became more complex all of that this started gathering a lot of data right.

Now to process the data the whole concept first structure the data and store the data the whole concept of Big Data came into play right and then as recently as five-six years ago people just figuring out on using Hadoop and data Lakes meters to store the data right but with the advent of such large volume of data, although then there was a need to use more sophisticated techniques which were more efficient in handling this kind of data and those techniques then were called data science right and then from their companies a move to using machine learning techniques and for some use cases.

Now a lot of companies have started using AI and deep learning so so this is the way you know this is the data maturity in adoption in companies this is the way it happens in each company and any company that you think of is lying somewhere or the other in this park I can right now most of the companies are at least between data science and machine learning right and many of those are had crossing machine learning point and then they are moving towards utilizing more declawing techniques Hey but even then even.

If the company is not developed to such an extent right it is the next logical step for that company so every company will have to go through this cycle will have to sorry for this path will have to develop and use more and more sophisticated techniques as they go along right because as I was saying it is not an option anymore in this competitive world right and the kind of use cases that we are seeing right now which are enabled through some of these advanced techniques, for example, I personally I used to work at a bank called Capital One it’s the fifth-largest bank in the US and I used to be a director there and I had you know a team.