Linear Regression with Single Variable (Univariate Linear Regression)
In these type of problems, we are given a set of data (training set) and looking at the data, we need to predict future set. For example, for a given training set consisting of house size and price, we need to find the price of a house with given size.
Mathematically, it can be represented as a linear equation:
h0(x) = 00 + 01x
We need choose 00 and 01 such that h0(x) is as close to Y as possible for our data set (variance should be minimal).
The function of h0(x) and Y is called Cost function. The minimum value of this cost function is desired i.e. all points stay close to the line on the graph.
In summary,
Cost Function
In order to understand cost function and plot its graph against 00 or 01, we can keep value of both parameters fixed.
Please pay attention to the graph on the right side. It is a plot of cost function w.r.t 01.
Now, we can expand our understanding by changing value of 01.
Gradient Decent
(one of the technique to determine values of Thetas)
Goal: Find minimum value of the cost function we defined earlier. In order to do so, find optimal value of 00 and 01.
In order to do so, start with some 00 and 01 and keep changing both values simultaneously so that we ultimately find minimum value of cost function.
Algorithm
- If learning rate (alpha) is too small, gradient descent can be slow.
- If alpha is too large, gradient descent can overshoot the minimum. If may fail to converge or even diverge.
- Gradient descent can converge to a local minimum, even when learning rate fixed.
Please remember that there could be multiple local minimum but we need to find a global minimum(lowest point or lowest height).
