안녕하세요, HELLO
오늘은 DeepLearning.AI에서 진행하는 앤드류 응(Andrew Ng) 교수님의 딥러닝 전문화의 두 번째 과정인 "Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization"을 정리하려고 합니다.
"Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization"의 강의 목적은 '랜덤 초기화, L2 및 드롭아웃 정규화, 하이퍼파라미터 튜닝, 배치 정규화 및 기울기 검사와 같은 표준 신경망 기술' 등을 배우며, 강의는 아래와 같이 구성되어 있습니다.
~ Practical Aspects of Deep Learning
~ Optimization Algorithms
~ Hyperparameter Tuning, Batch Normalization and Programming Frameworks
"Improving Deep Neural Networks" (Andrew Ng)의 1주차 "Gradient Checking"의 실습 내용입니다.
- Implement gradient checking to verify the accuracy of your backprop implementation
CHAPTER 1. 'Packages'
CHAPTER 2. 'Problem Statement'
CHAPTER 3. 'How does Gradient Checking work?'
CHAPTER 4. '1-Dimensional Gradient Checking'
CHAPTER 5. 'N-Dimensional Gradient Checking'
CHAPTER 1. 'Packages'
이번 실습에서 사용되는 패키지, 라이브러리입니다.
import numpy as np
from testCases import *
from public_tests import *
from gc_utils import sigmoid, relu, dictionary_to_vector, vector_to_dictionary, gradients_to_vector
%load_ext autoreload
%autoreload 2
CHAPTER 2. 'Problem Statement'
You are part of a team working to make mobile payments available globally, and are asked to build a deep learning model to detect fraud--whenever someone makes a payment, you want to see if the payment might be fraudulent, such as if the user's account has been taken over by a hacker.
You already know that backpropagation is quite challenging to implement, and sometimes has bugs. Because this is a mission-critical application, your company's CEO wants to be really certain that your implementation of backpropagation is correct. Your CEO says, "Give me proof that your backpropagation is actually working!" To give this reassurance, you are going to use "gradient checking."
CHAPTER 3. 'How does Gradient Checking work?'
Backpropagation computes the gradients ∂𝐽∂𝜃, where 𝜃 denotes the parameters of the model. 𝐽 is computed using forward propagation and your loss function.
Because forward propagation is relatively easy to implement, you're confident you got that right, and so you're almost 100% sure that you're computing the cost 𝐽 correctly. Thus, you can use your code for computing 𝐽 to verify the code for computing ∂𝐽∂𝜃
CHAPTER 4. '1-Dimensional Gradient Checking'
Consider a 1D linear function 𝐽(𝜃)=𝜃𝑥. The model contains only a single real-valued parameter 𝜃, and takes 𝑥 as input.
You will implement code to compute 𝐽(.) and its derivative ∂𝐽∂𝜃. You will then use gradient checking to make sure your derivative computation for 𝐽J is correct.
□ Forward progation
# GRADED FUNCTION: forward_propagation
def forward_propagation(x, theta):
"""
Implement the linear forward propagation (compute J) presented in Figure 1 (J(theta) = theta * x)
Arguments:
x -- a real-valued input
theta -- our parameter, a real number as well
Returns:
J -- the value of function J, computed using the formula J(theta) = theta * x
"""
# (approx. 1 line)
# J =
# YOUR CODE STARTS HERE
J = theta * x
# YOUR CODE ENDS HERE
return J
x, theta = 2, 4
J = forward_propagation(x, theta)
print ("J = " + str(J))
forward_propagation_test(forward_propagation)
# J = 8□ Backward propagation
Now, implement the backward propagation step (derivative computation) of Figure 1. That is, compute the derivative of 𝐽(𝜃)=𝜃𝑥 with respect to 𝜃. To save you from doing the calculus, you should get 𝑑𝑡ℎ𝑒𝑡𝑎=∂𝐽, ∂𝜃=𝑥.
# GRADED FUNCTION: backward_propagation
def backward_propagation(x, theta):
"""
Computes the derivative of J with respect to theta (see Figure 1).
Arguments:
x -- a real-valued input
theta -- our parameter, a real number as well
Returns:
dtheta -- the gradient of the cost with respect to theta
"""
# (approx. 1 line)
# dtheta =
# YOUR CODE STARTS HERE
dtheta = x
# YOUR CODE ENDS HERE
return dtheta
x, theta = 2, 4
dtheta = backward_propagation(x, theta)
print ("dtheta = " + str(dtheta))
backward_propagation_test(backward_propagation)
# dtheta = 2□ Gradient check
Instructions:
- First compute "gradapprox" using the formula above (1) and a small value of 𝜀ε. Here are the Steps to follow:
- 𝜃+=𝜃+𝜀
- 𝜃−=𝜃−𝜀
- 𝐽+=𝐽(𝜃+)
- 𝐽−=𝐽(𝜃−)
- 𝑔𝑟𝑎𝑑𝑎𝑝𝑝𝑟𝑜𝑥=(𝐽+−𝐽) /2𝜀
- Then compute the gradient using backward propagation, and store the result in a variable "grad"
- Finally, compute the relative difference between "gradapprox" and the "grad" using the following formula:
You will need 3 Steps to compute this formula:
1'. compute the numerator using np.linalg.norm(...)
2'. compute the denominator. You will need to call np.linalg.norm(...) twice.
3'. divide them.
If this difference is small (say less than 10−7), you can be quite confident that you have computed your gradient correctly. Otherwise, there may be a mistake in the gradient computation.
# GRADED FUNCTION: gradient_check
def gradient_check(x, theta, epsilon=1e-7, print_msg=False):
"""
Implement the backward propagation presented in Figure 1.
Arguments:
x -- a float input
theta -- our parameter, a float as well
epsilon -- tiny shift to the input to compute approximated gradient with formula(1)
Returns:
difference -- difference (2) between the approximated gradient and the backward propagation gradient. Float output
"""
# Compute gradapprox using left side of formula (1). epsilon is small enough, you don't need to worry about the limit.
# (approx. 5 lines)
# theta_plus = # Step 1
# theta_minus = # Step 2
# J_plus = # Step 3
# J_minus = # Step 4
# gradapprox = # Step 5
# YOUR CODE STARTS HERE
theta_plus = theta + epsilon
theta_minus = theta - epsilon
J_plus = forward_propagation(x, theta_plus)
J_minus = forward_propagation(x, theta_minus)
gradapprox = (J_plus - J_minus) / (2 * epsilon)
# YOUR CODE ENDS HERE
# Check if gradapprox is close enough to the output of backward_propagation()
#(approx. 1 line) DO NOT USE "grad = gradapprox"
# grad =
# YOUR CODE STARTS HERE
grad = backward_propagation(x, theta)
# YOUR CODE ENDS HERE
#(approx. 1 line)
# numerator = # Step 1'
# denominator = # Step 2'
# difference = # Step 3'
# YOUR CODE STARTS HERE
numerator = np.linalg.norm(grad-gradapprox)
demoniator = np.linalg.norm(grad) + np.linalg.norm(gradapprox)
difference = numerator / demoniator
# YOUR CODE ENDS HERE
if print_msg:
if difference > 2e-7:
print ("\033[93m" + "There is a mistake in the backward propagation! difference = " + str(difference) + "\033[0m")
else:
print ("\033[92m" + "Your backward propagation works perfectly fine! difference = " + str(difference) + "\033[0m")
return difference
x, theta = 2, 4
difference = gradient_check(2,4, print_msg=True)
#gradient_check_test(gradient_check)
# Your backward propagation works perfectly fine! difference = 2.919335883291695e-10
Congrats, the difference is smaller than the 10−7 threshold. So you can have high confidence that you've correctly computed the gradient in backward_propagation().
Now, in the more general case, your cost function 𝐽J has more than a single 1D input. When you are training a neural network, 𝜃 actually consists of multiple matrices 𝑊[𝑙] and biases 𝑏[𝑙]. It is important to know how to do a gradient check with higher-dimensional inputs. Let's do it!
CHAPTER 5. 'N-Dimensional Gradient Checking'
The following figure describes the forward and backward propagation of your fraud detection model.
def forward_propagation_n(X, Y, parameters):
"""
Implements the forward propagation (and computes the cost) presented in Figure 3.
Arguments:
X -- training set for m examples
Y -- labels for m examples
parameters -- python dictionary containing your parameters "W1", "b1", "W2", "b2", "W3", "b3":
W1 -- weight matrix of shape (5, 4)
b1 -- bias vector of shape (5, 1)
W2 -- weight matrix of shape (3, 5)
b2 -- bias vector of shape (3, 1)
W3 -- weight matrix of shape (1, 3)
b3 -- bias vector of shape (1, 1)
Returns:
cost -- the cost function (logistic cost for one example)
cache -- a tuple with the intermediate values (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3)
"""
# retrieve parameters
m = X.shape[1]
W1 = parameters["W1"]
b1 = parameters["b1"]
W2 = parameters["W2"]
b2 = parameters["b2"]
W3 = parameters["W3"]
b3 = parameters["b3"]
# LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID
Z1 = np.dot(W1, X) + b1
A1 = relu(Z1)
Z2 = np.dot(W2, A1) + b2
A2 = relu(Z2)
Z3 = np.dot(W3, A2) + b3
A3 = sigmoid(Z3)
# Cost
log_probs = np.multiply(-np.log(A3),Y) + np.multiply(-np.log(1 - A3), 1 - Y)
cost = 1. / m * np.sum(log_probs)
cache = (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3)
return cost, cache
def backward_propagation_n(X, Y, cache):
"""
Implement the backward propagation presented in figure 2.
Arguments:
X -- input datapoint, of shape (input size, 1)
Y -- true "label"
cache -- cache output from forward_propagation_n()
Returns:
gradients -- A dictionary with the gradients of the cost with respect to each parameter, activation and pre-activation variables.
"""
m = X.shape[1]
(Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cache
dZ3 = A3 - Y
dW3 = 1. / m * np.dot(dZ3, A2.T)
db3 = 1. / m * np.sum(dZ3, axis=1, keepdims=True)
dA2 = np.dot(W3.T, dZ3)
dZ2 = np.multiply(dA2, np.int64(A2 > 0))
dW2 = 1. / m * np.dot(dZ2, A1.T)
db2 = 1. / m * np.sum(dZ2, axis=1, keepdims=True)
dA1 = np.dot(W2.T, dZ2)
dZ1 = np.multiply(dA1, np.int64(A1 > 0))
dW1 = 1. / m * np.dot(dZ1, X.T)
db1 = 1. / m * np.sum(dZ1, axis=1, keepdims=True)
gradients = {"dZ3": dZ3, "dW3": dW3, "db3": db3,
"dA2": dA2, "dZ2": dZ2, "dW2": dW2, "db2": db2,
"dA1": dA1, "dZ1": dZ1, "dW1": dW1, "db1": db1}
return gradients□ How does gradient checking work?.
As in Section 3 and 4, you want to compare "gradapprox" to the gradient computed by backpropagation. The formula is still:
However, 𝜃θ is not a scalar anymore. It is a dictionary called "parameters". The function "dictionary_to_vector()" has been implemented for you. It converts the "parameters" dictionary into a vector called "values", obtained by reshaping all parameters (W1, b1, W2, b2, W3, b3) into vectors and concatenating them.
The inverse function is "vector_to_dictionary" which outputs back the "parameters" dictionary.
The "gradients" dictionary has also been converted into a vector "grad" using gradients_to_vector(), so you don't need to worry about that.
Now, for every single parameter in your vector, you will apply the same procedure as for the gradient_check exercise. You will store each gradient approximation in a vector gradapprox. If the check goes as expected, each value in this approximation must match the real gradient values stored in the grad vector.
Note that grad is calculated using the function gradients_to_vector, which uses the gradients outputs of the backward_propagation_n function.
□ Gradient_check_n
you get a vector gradapprox, where gradapprox[i] is an approximation of the gradient with respect to parameter_values[i]. You can now compare this gradapprox vector to the gradients vector from backpropagation. Just like for the 1D case (Steps 1', 2', 3'), compute:
# GRADED FUNCTION: gradient_check_n
def gradient_check_n(parameters, gradients, X, Y, epsilon=1e-7, print_msg=False):
"""
Checks if backward_propagation_n computes correctly the gradient of the cost output by forward_propagation_n
Arguments:
parameters -- python dictionary containing your parameters "W1", "b1", "W2", "b2", "W3", "b3":
grad -- output of backward_propagation_n, contains gradients of the cost with respect to the parameters.
x -- input datapoint, of shape (input size, 1)
y -- true "label"
epsilon -- tiny shift to the input to compute approximated gradient with formula(1)
Returns:
difference -- difference (2) between the approximated gradient and the backward propagation gradient
"""
# Set-up variables
parameters_values, _ = dictionary_to_vector(parameters)
grad = gradients_to_vector(gradients)
num_parameters = parameters_values.shape[0]
J_plus = np.zeros((num_parameters, 1))
J_minus = np.zeros((num_parameters, 1))
gradapprox = np.zeros((num_parameters, 1))
# Compute gradapprox
for i in range(num_parameters):
# Compute J_plus[i]. Inputs: "parameters_values, epsilon". Output = "J_plus[i]".
# "_" is used because the function you have to outputs two parameters but we only care about the first one
#(approx. 3 lines)
# theta_plus = # Step 1
# theta_plus[i] = # Step 2
# J_plus[i], _ = # Step 3
# YOUR CODE STARTS HERE
theta_plus = np.copy(parameters_values)
theta_plus[i][0] = theta_plus[i][0] + epsilon
J_plus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(theta_plus))
# YOUR CODE ENDS HERE
# Compute J_minus[i]. Inputs: "parameters_values, epsilon". Output = "J_minus[i]".
#(approx. 3 lines)
# theta_minus = # Step 1
# theta_minus[i] = # Step 2
# J_minus[i], _ = # Step 3
# YOUR CODE STARTS HERE
theta_minus = np.copy(parameters_values)
theta_minus[i][0] = theta_minus[i][0] - epsilon
J_minus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(theta_minus))
# YOUR CODE ENDS HERE
# Compute gradapprox[i]
# (approx. 1 line)
# gradapprox[i] =
# YOUR CODE STARTS HERE
gradapprox[i] = (J_plus[i] - J_minus[i]) / (2 * epsilon)
# YOUR CODE ENDS HERE
# Compare gradapprox to backward propagation gradients by computing difference.
# (approx. 1 line)
# numerator = # Step 1'
# denominator = # Step 2'
# difference = # Step 3'
# YOUR CODE STARTS HERE
numerator = np.linalg.norm(grad - gradapprox)
demoniator = np.linalg.norm(grad) + np.linalg.norm(gradapprox)
difference = numerator / demoniator
# YOUR CODE ENDS HERE
if print_msg:
if difference > 2e-7:
print ("\033[93m" + "There is a mistake in the backward propagation! difference = " + str(difference) + "\033[0m")
else:
print ("\033[92m" + "Your backward propagation works perfectly fine! difference = " + str(difference) + "\033[0m")
return difference
X, Y, parameters = gradient_check_n_test_case()
cost, cache = forward_propagation_n(X, Y, parameters)
gradients = backward_propagation_n(X, Y, cache)
difference = gradient_check_n(parameters, gradients, X, Y, 1e-7, True)
expected_values = [0.2850931567761623, 1.1890913024229996e-07]
assert not(type(difference) == np.ndarray), "You are not using np.linalg.norm for numerator or denominator"
assert np.any(np.isclose(difference, expected_values)), "Wrong value. It is not one of the expected values"
# Your backward propagation works perfectly fine! difference = 1.1890913024229996e-07□ What you should remember from this notebook:
- Gradient checking verifies closeness between the gradients from backpropagation and the numerical approximation of the gradient (computed using forward propagation).
- Gradient checking is slow, so you don't want to run it in every iteration of training. You would usually run it only to make sure your code is correct, then turn it off and use backprop for the actual learning process.
■ 마무리
"Improving Deep Neural Networks" (Andrew Ng)의 1주차 "Gradient Checking"의 실습에 대해서 정리해봤습니다.
그럼 오늘 하루도 즐거운 나날 되길 기도하겠습니다
좋아요와 댓글 부탁드립니다 :)
감사합니다.
'COURSERA' 카테고리의 다른 글
| week 2_Optimization Algorithms 연습 문제 (Andrew Ng) (0) | 2022.02.20 |
|---|---|
| week 2_Optimization Algorithms (Andrew Ng) (0) | 2022.02.20 |
| week 1_Regularization 실습 (Andrew Ng) (0) | 2022.02.18 |
| week 1_Initialization 실습 (Andrew Ng) (0) | 2022.02.18 |
| week 1_Practical Aspects of Deep Learning 연습문제 (Andrew Ng) (0) | 2022.02.18 |
댓글