Transcripts
1. Introduction to Lesson 2: Hi, thanks for clicking on this A0 product management
course. My name is Kristin. I have been working in
IT almost ten years and in data science and
computer vision, almost five. Hi, I'm Lizzie and
welcome to the course. I have also included in AI product development
in the past three years, been working on automating
verification process with machine learning and specifically
with computer vision. In this course, we will go more in-depth into AI
product development. We'll discuss the
differences between a software development mindset and AI product
development mindset. Also, we will look
why AI products take so much time for maintenance and why
they degrade over time. The last thing for this course, we will go and look into the three steps that
you need to make sure and start thinking about when creating
your first Ai project. Thank you.
2. Mindset shift in AI development: In the previous
lesson, we talked about the fact that there is a computer's
ability to perform tasks that are usually
done by people. Most AI applications work
with really specific tasks. Ai development and software development processes
are very different. We will be discussing that more in depth this
lesson as well. We talked about AI
product requirements and about what they depend on. And we talked about AI products and what sort of AI
today actually include. Today's topic is what sorts
of work to actually live in. In terms of data. It is sometimes are
good whether we live in a world where everything
is destined to be, or in a world where we create their own future with
our own decisions. We understand that
there are tasks that lead to a specific outcome and there are
decisions that lead to a surprise or an
unexpected outcome. This world can be described as a deterministic and
probabilistic world. The term deterministic
world means we know what will happen
if you do one thing, other thing will
definitely happen. Classical software
engineering problems are just like that. Deterministic. If you click on a button, it does something it
is programmed to do. I mean, of course
unless there is a bug. But when we're talking about AI products or NEA
functionality, we are leaning more
into non-deterministic, also called probabilistic world. This means that we cannot be
sure what the model will do. This is specifically
important to understand when mothers are put into an unknown situation. For example, when
the input data is slightly different
from the regular norm. In computer vision, this
would mean that the image includes objects or environments that the model is not used to. So let's explain
this in more detail. If some activity
happens, for example, press of a button or entering a query or receiving
data in a form. There is always some sort of concrete functions
related to that. You click on a button,
next page is opened, or query is entered, and then the result is
searched for and database. Or third example, data is received and it is automatically
saved into database. Although things
are deterministic. But then click
happened. Click equals true and something
predefined is then done. But what about
probabilistic world? There is a balance of
statistics and guesses. So we do not exactly know for sure that
something will happen, but we can predict it based
on previous patterns. For example, if you see a
dog player barking sound, or in fraud detection, if it seems, or fields that
user X strangely then sent, sent them for
further inspection. The main difference with deterministic world
is that we kinda be 100% sure that anything
happened or will happen. It is not like there may be, what's a click of a button, let's send the data anyway. But our AI algorithms, deterministic or probabilistic,
we can say that they are, is deterministic
in the sense that does follow the
algorithm or the model, but probabilistic in the sense of creating that algorithm. So when we are training a model, it uses statistics to predict the outcome, meaning
it's probabilistic. But when this train
model is then put into use in a product, it predicts exactly
by the rules or the, or the algorithm
is trained to do. We can also have a quick look into deterministic
and probabilistic data. Probabilistic data is
data that is based on behavioral events
like pageviews, time spent on page
are click-throughs. This sort of data can be
used for prediction models. For example, it can be
used for clustering. Users are suggesting
items to your customer. Deterministic data is linked to something which
identifies the user, like an email address
or a cookie idea id. And it's most likely 100% true. This can be considered a fact or a
characteristic of a user. For example, if a
user signs up to some platform and gives
their current age, it is, it is a fact
that the following year the customer
will be a year older. But the, but if the same user use the
platform ten times less tear, we cannot be sure if he or she will use the same
platform next year, 1110 or nine times. Whilst we can do is group and analyze many users
with similar behavior last year and make a guess or prediction how he or she
might deck next year. But we don't know it for sure. So when we're talking
about mindset, when creating our products, we have to understand that the real world is
not deterministic. Real-world is not
black and white. It's, we can say, Fifty Shades of Grey. So we leave in a probabilistic
world and we have to make all those
predictions based on that, AI is solving
probabilistic challenges. When a great day or products, We need to make sense of the probabilistic
world and structure it into data that describes the world around us in a
way that makes sense to AI. In last lesson, we discussed
the difference of processes between software engineering
and product development. But how does this probabilistic world
influence development in more detail? So for software
engineering products, there is a high likelihood
of knowing the outcome. Development time can be
estimated quite well. People use Scrum
processes and other stuff to estimate more better. The goal can be described
by mockups, specs analysis. There are established
best practices of the engineering process. And product problems do not
really shift that often. And when the feature is ready, it's mostly, it most likely stays like this for a long time. But when we're talking
about AI products, we cannot be sure
of anything really. There was a hope for an outcome. You want the model
to do something, but you cannot be sure if
it will actually do it. You cannot really estimate, but can timebox tasks. So it's more or less unknown how long it will
take to reinvent the bike. So there's lot of
guessing happening. The goal of the
model is described through data examples
and metrics. There are not yet established best practices about managing
day AI development process. So we can say it's
a bit of a mess. And the environment that
the data, the models, or the AI features
use the change really often. The
model is awesome. It costs constant updates. We'd have to keep maintaining
them for a long time.
3. Consistent Maintenance - Why?: And systems maintenance and
why it's really necessary for AI products is really depending on where the
product is applied to. What are the environments
that it's applied to. Because if you apply the model to a different place that
it wasn't trained for, there is an environmental shift and you need to do
updates for it. Also, when environment
around us changes, then also the data and
the patterns change. If life changes,
then basically we need to change the model, the product with it. Let's take an example of corona. When Corona started, people
started wearing face masks. If you had face detection model that tried to detect faces. Now the environment
changes, changed. If you had lots of people
with face masks before, you will have anybody
basically with face masks. New clients. If you are in a place where you apply your model to different clients,
different environments. For example, first you
have a client in us, then you have client in Europe. Than there are different
patterns of human behavior on the environment and how
everything works looks like. So there is a data shift
through with that. Also, we are doing constant
maintenance because machine learning is moving
around really fast. It's growing year by year. And some other
things you might not be so couldn't sold last year. You can solve this
year and therefore, you just need to come back
to apply new methods. When you build a
software product, the environment you
solve the problem is usually way more stable
than for AI products. At least it doesn't show up so easily when you
create a website, you usually, for example, have understanding who
is the main clients? Is it for elderly people, is it for children? You probably will do some changes depending
on who this product is meant to fool the AI
product developments. Yes, you're also thinking
with this applies, but there are a lot of more
different factors that play in the performance
of these models. Ar developments are specifically optimizing only
really small problem set if problems and changes
to performance also changes. Let's take an example. So Forest animal
detection algorithm, the aim of it is that we detect animals and we classify
what animal it is. So here you can see the
example. There is rapid. We say it's rabbit. If it has do with the
stair and say what it is. Now, we are in Northern Europe. This is where we are
building our animals, but there are 20
animals here in total. So what we are going to first
focus on when trying to build a good machine
learning model is take the biggest
group that we can do. The biggest group today is that they're so thick that
they are correctly. We can detect seven
animals after 20. Next one after that is born. So we have four boards and
if we detect them correctly, we can already addicted
for more animals. After that, there is foxes. So we can catch three
folks is awesome. So this is kinda the
distribution of the environment. With this environment
and with this model, we catch 70% of the animals. So with detecting deer
meat, the ticking boxes, with the dictum for now, we have a change that
somebody has decided that we should put the same model and
use it in tropical forest. So what we will do is that
there is a new environment. We have exactly the same model. So everything is the same. But what we optimize
it for is different. So first we will look at dear, oh, there's only one there. So we catch it only one animal. What other thing that we
learned how to the deck? So they were the board, so we have three boards. They take the dose also
that they are pores. Next we detected foxes, at least that's what
the model I learned. There is no boxes detected. So we lost that
class completely. So here you can see the
change of environment. Really clearly. We changed where we're using the ER and the ER
doesn't understand it. So in tropical forest, the algorithm catches 20%
of the animals overall. So we cannot state that
the algorithm is bad. It's more than where we are applying it and what
it was meant to be. This is really drastic
change in real-life. Lot of the times we don't
have so drastic changes. But there are changes up to
10% and even smaller changes, but they bring up the problem
of fairness and bias. So we're all for
machine learning.
4. AI product development: We'll go into product
development part. So first is the three pillars to think before you start
working on in our predicts. So really have to think about
those because otherwise you cannot be successful with
solving the problem or, or creating the product. So first, what problem
you're trying to solve? What are the things
you're trying to solve? What are the things that we are fixing here
are making better? Second, what metrics
you care about? Really, what are the
things you care about? And we'll go into more details. You will see the examples. And thirdly, where
do you get the data? In the previous course, we looked at the animals. You saw the data is really
important and you have to keep in mind where
do we get the data? Because the data
that gets really determines how well you perform on the metrics
that you care about. There are two kinds of
problems or overshoot. Basically, it's all
with machine learning. One is automating the process, another is estimating
an outcome. In automating the process, we mean basically
automating humans work, factory work,
customers report work. Really today, a work that
is really repetitive. We are good at narrow
AI today in the world, not with general ARE. So we can automate small tasks. Estimating an outcome is
also something that is done. It's for example,
when you want to estimate what people
like, like marketing. Or you want to estimate
how much time it takes to get from one place to
another place and tomorrow, because tomorrow is Monday. So something like that. Estimation and automation,
the aim of them is different, but methods behind is our same. Automation. Automation
is, like I said, as an example, it's a factory work guides
repetitive tasks. How to understand what
task job to meet. There could be also
really hard tasks that are done in factors
or the derivative. Some of them could be a, teach a child how to read. It's repetitive tasks. Teachers are doing
it all the time, but it's hard tasks. But what machine learning
can do in the ER can do is more small tasks,
1 second tasks. So start thinking of your
problem as 1 second tasks. What are the things you
can do in 1 second? And this is one task. This is one problem to fix. If you have a bigger problem, you can split it into
different 1 second tasks. For example, if you want to, um, to a robot that finds
garbage and throws it away, then first is look around. Second, detect garbage. Third, go near the garbage. Four. Take it up, go on and go
on and then throw it away. So you can see they're
really small task. You cannot have a task
clean up the room. But easier to task, the more likely
it will get done. So all of us understand that. And also the ones that control is kinda guidance to
understand what are the things that can be done
or how it should be splitted? If the task is basically
a really small, if it's real easy task, it's really easy to solve it if there is lot of details of task also a lot of things to
keep in mind to do roles. There is way harder. So really, really simple things that
we can do these days to do. And your aim as a product
manager should be think of a tasks so that they
won't be atlas T2. This means it's easier
for you to solve it. Estimations, estimation task
is what people are like, would like estimate that, um, and this is basically that today it's used mostly for marketing purposes. And to basically in the
consumer, consumer side. Secondly, what kind of
metrics you care about? The big, big thing
here is to forget to always be a 100% correct. You can never be 100% correct. People can never
be a 100% correct. Nor can AI. Today is, like I said, once, to solve really
small tasks things. So the bigger question
to answer is, what mistakes are you
willing to make water and do things that are
okay to mistake up? Let's look at some examples. For example, rare condition in a shop. So we are in a show. It's basically the estimation
you tried to understand. People like that. Or what you can do is show
glows clients lengths. That's really good.
Don't show gloss. Client does not lie. Meaning you don't show
things that they don't like. They are really
good things to do. One of the things you could do, your show glows that
clients do not like. So this means that you show clients things that
for them is garbage. They, they see something
they don't care about. Or also what you
can do as mistake is don't show clothes
that the client likes. That means that you don't show the things
that they really like. And you, as a marketer will
have losing revenue because you don't sell tightens
that the client would buy. Also, another example is good. So good rates, you know, is a recommendation for books. So here is recommendations made and some of
the things maybe, maybe like Salton things. Not. Fourthly recommended is a
book person does not lie and forcefully not recommended
a book person likes. So these are the two mistakes
that you can make today. So whether you do
not show books, people like, or you basically show them
stuff that they don't curb. Looking at this system today. Probably if you would get load of books that are
recommended to you, but you don't like them, then you're not going to
use that service anymore because you see that the
recommendation is pointless. Additional, good.
What is more okay? Is not recommending a
burst on the books. People like recommending
most of them. So here you can see why
the first one is not okay. But the second one is because when the first thing
happened all the time, then the client would
not use the service. But if the second one happens, then Pip probably people
will still line to service. They just don't. We'll
get all the books that would be recommended
or good for them. Another example on the planes. Planes have lot of maintenance. They need a lot of repairs
done them and they need to follow a lot of
requirements to flying. Also, terrorists, basically similar
calculations behind it. Falsely state that one piece
of the airplane is not okay. Or falsely state that
the piece is okay. So if we are the
first and we say falsely that one piece of
equipment is not okay. We just use more time
to repair the plane. We use more money to
repair the plane. But nothing bad happens. If a four-fifths state
that a piece is okay. This means that we falsely state that that piece is your
k and let the plane fly, we will get a faulty plane. So today, in this case, this is not a good thing. Something bad might happen with a plane when we let it fly. So you could see
before that all of those things are
depending use case. Let's go on to what
kind of data you have. It this hard restored
renewal data. First, you always have
to have some things, but this is a clear thing. It's really, really, really hard to start from
completely topic. But what kind of data you
need is also be questioned. And that now depends
on the problem that you have written down. You need data that
is as close as possible to the data
you have in real life. Best would be that you
have data of really, really the live, where
you would use it. Coming back to the forest
example with animals, if you will use the animal
detector in tropical forest, it's best to have the data
room tropical forest, not from some other
kind of forest. So really data from exactly where you will use
and the product. Let's take an example
of a car detector. You want AI to know what
is a car and what is not. So the question
that we learn is, what is, is it a car? So the left side, it is the right side. It's not, it's a Googler round. In search for cars, you will see tons of
different cars around. What you can see here is
first different color. Some are in white background,
some are driving, some are in Lakish
grayish background. This is completely
from the side. This is more on the top and
not completely at the top. This is from the front. So if you do a car detector, what side you want
to see the car? And is it on the white or black background
or the elements? Or is it in in real life, in near the beach
or on the street? Do you need to
detect cars on here, these pictures are
really different, the ones that we had just
in the previous slide. Or do you want to to see
and detect cars from above? We didn't have the any
cost pictures from above and the picture above
is completely different. So you really need to think from what angle you also
want to detect them. Because if you train a model to detect the cars from side, but in reality the camera of
where you in the future with detector cars are
looking at from the top, the model will not perform
AN will fail here. So the data has to be as similar
to the use case that you have in real life from geographical
lightning perspective. So we sit in a dark room,
light room, time-wise. I'm 50-year-old cars. Cars right now, streets 50 years ago or today. Maybe. Things don't change
so much lately and you don't have so much pictures
from fifties. But basically, there are every now and then there's new
things happening. So time is really
important to get the latest data and
viewpoints and angles. So making sure that
you are looking at objects as similar way
as you in the future. One to have the use case for.
5. Conclusion: For me, it's like stated above, the AI product
performance depends hugely on the available
data that you have in. Because the data that comes in is basically performance
that comes out. So what you are
collecting that you are getting also from the
client data or real you, where you are using the data. This is basically
determined the off towards performance of
this model because this depends where
it is applied to. So where is the real use case? And it is hugely important
that the data that you have in the usage decline data and
available data you have when training or as
similar as possible. In the conclusion, for
this day's course, development verbs together
with software development, there is nothing separating those 28 development is riskier, known, unknown, a
lot unknown than so for development because
it's research and development. And also there is a
lot of dependencies of way you're going
to use that model. Development is dependent. Children data. They could say it
or sore, ready. The data, what you're
using for training, How has to be really similar to where are you going to
use it in the future. As did data determines also
how will you will perform. Ei prefer forms is stepping link hugely
on the environment. So data and environment. They earned Kiki, thanks. Thank you for listening today. See you next time.
Liisi Soots, AI Product Development
