728x90
반응형
안녕하세요, HELLO
오늘은 DeepLearning.AI에서 진행하는 앤드류 응(Andrew Ng) 교수님의 딥러닝 전문화의 마지막이며, 다섯 번째 과정인 "Sequence Models"을 정리하려고 합니다.
"Sequence Models"의 강의를 통해 '시퀀스 모델과 음성 인식, 음악 합성, 챗봇, 기계 번역, 자연어 처리(NLP)' 등을 이해하고, 순환 신경망(RNN)과 GRU 및 LSTM, 트랜스포머 모델에 대해서 배우게 됩니다. 강의는 아래와 같이 구성되어 있습니다.
~ Recurrent Neural Networks
~ Natural Language Processing & Word Embeddings
~ Sequence Models & Attention Mechanism
~ Transformer Network
"Sequence Models" (Andrew Ng)의 2주차 "Emojify"의 실습 내용입니다.
Using Word Vectors to Improve Emoji Lookups
- In many emoji interfaces, you need to remember that ❤️ is the "heart" symbol rather than the "love" symbol.
- In other words, you'll have to remember to type "heart" to find the desired emoji, and typing "love" won't bring up that symbol.
- You can make a more flexible emoji interface by using word vectors!
- When using word vectors, you'll see that even if your training set explicitly relates only a few words to a particular emoji, your algorithm will be able to generalize and associate additional words in the test set to the same emoji.
- This works even if those additional words don't even appear in the training set.
- This allows you to build an accurate classifier mapping from sentences to emojis, even using a small training set.
What you'll build:
- In this exercise, you'll start with a baseline model (Emojifier-V1) using word embeddings.
- Then you will build a more sophisticated model (Emojifier-V2) that further incorporates an LSTM.
By the end of this notebook, you'll be able to:
- Create an embedding layer in Keras with pre-trained word vectors
- Explain the advantages and disadvantages of the GloVe algorithm
- Describe how negative sampling learns word vectors more efficiently than other methods
- Build a sentiment classifier using word embeddings
- Build and train a more sophisticated classifier using an LSTM
CHAPTER 1. 'Packages'
CHAPTER 2. 'Baseline Model: Emojifier-V1'
CHAPTER 3. 'Emojifier-V2: Using LSTMs in Keras'
CHAPTER 1. 'Packages'
import numpy as np
from emo_utils import *
import emoji
import matplotlib.pyplot as plt
from test_utils import *
%matplotlib inline
CHAPTER 2. 'Baseline Model: Emojifier-V1'
□ Dataset EMOJISET
You have a tiny dataset (X, Y) where:
- X contains 127 sentences (strings).
- Y contains an integer label between 0 and 4 corresponding to an emoji for each sentence.
X_train, Y_train = read_csv('data/train_emoji.csv')
X_test, Y_test = read_csv('data/tesss.csv')
maxLen = len(max(X_train, key=len).split())□ Overview of the Emojifier-V1
Inputs and Outputs
- The input of the model is a string corresponding to a sentence (e.g. "I love you").
- The output will be a probability vector of shape (1,5), (indicating that there are 5 emojis to choose from).
- The (1,5) probability vector is passed to an argmax layer, which extracts the index of the emoji with the highest probability.
One-hot Encoding
- To get your labels into a format suitable for training a softmax classifier, convert 𝑌 from its current shape (𝑚,1) into a "one-hot representation" (𝑚,5),
- Each row is a one-hot vector giving the label of one example.
- Here, Y_oh stands for "Y-one-hot" in the variable names Y_oh_train and Y_oh_test:
Y_oh_train = convert_to_one_hot(Y_train, C = 5)
Y_oh_test = convert_to_one_hot(Y_test, C = 5)□ Implementing Emojifier-V1
As shown in Figure 2 (above), the first step is to:
- Convert each word in the input sentence into their word vector representations.
- Take an average of the word vectors.
Similar to this week's previous assignment, you'll use pre-trained 50-dimensional GloVe embeddings.
Run the following cell to load the word_to_vec_map, which contains all the vector representations.
word_to_index, index_to_word, word_to_vec_map = read_glove_vecs('data/glove.6B.50d.txt')
You've loaded:
- word_to_index: dictionary mapping from words to their indices in the vocabulary
- (400,001 words, with the valid indices ranging from 0 to 400,000)
- index_to_word: dictionary mapping from indices to their corresponding words in the vocabulary
- word_to_vec_map: dictionary mapping words to their GloVe vector representation.
■ sentence_to_avg
# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: sentence_to_avg
def sentence_to_avg(sentence, word_to_vec_map):
"""
Converts a sentence (string) into a list of words (strings). Extracts the GloVe representation of each word
and averages its value into a single vector encoding the meaning of the sentence.
Arguments:
sentence -- string, one training example from X
word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation
Returns:
avg -- average vector encoding information about the sentence, numpy-array of shape (J,), where J can be any number
"""
# Get a valid word contained in the word_to_vec_map.
any_word = list(word_to_vec_map.keys())[0]
### START CODE HERE ###
# Step 1: Split sentence into list of lower case words (≈ 1 line)
words = sentence.lower().split()
# Initialize the average word vector, should have the same shape as your word vectors.
avg = np.zeros(word_to_vec_map[any_word].shape)
# Initialize count to 0
count = 0
# Step 2: average the word vectors. You can loop over the words in the list "words".
for w in words:
# Check that word exists in word_to_vec_map
if w in word_to_vec_map:
avg += word_to_vec_map[w]
# Increment count
count +=1
if count > 0:
# Get the average. But only if count > 0
avg = avg / count
### END CODE HERE ###
return avg
# BEGIN UNIT TEST
avg = sentence_to_avg("Morrocan couscous is my favorite dish", word_to_vec_map)
print("avg = \n", avg)
def sentence_to_avg_test(target):
# Create a controlled word to vec map
word_to_vec_map = {'a': [3, 3], 'synonym_of_a': [3, 3], 'a_nw': [2, 4], 'a_s': [3, 2],
'c': [-2, 1], 'c_n': [-2, 2],'c_ne': [-1, 2], 'c_e': [-1, 1], 'c_se': [-1, 0],
'c_s': [-2, 0], 'c_sw': [-3, 0], 'c_w': [-3, 1], 'c_nw': [-3, 2]
}
# Convert lists to np.arrays
for key in word_to_vec_map.keys():
word_to_vec_map[key] = np.array(word_to_vec_map[key])
avg = target("a a_nw c_w a_s", word_to_vec_map)
assert tuple(avg.shape) == tuple(word_to_vec_map['a'].shape), "Check the shape of your avg array"
assert np.allclose(avg, [1.25, 2.5]), "Check that you are finding the 4 words"
avg = target("love a a_nw c_w a_s", word_to_vec_map)
assert np.allclose(avg, [1.25, 2.5]), "Divide by count, not len(words)"
avg = target("love", word_to_vec_map)
assert np.allclose(avg, [0, 0]), "Average of no words must give an array of zeros"
avg = target("c_se foo a a_nw c_w a_s deeplearning c_nw", word_to_vec_map)
assert np.allclose(avg, [0.1666667, 2.0]), "Debug the last example"
print("\033[92mAll tests passed!")
sentence_to_avg_test(sentence_to_avg)
# END UNIT TEST
□ Implement the Model
You now have all the pieces to finish implementing the model() function! After using sentence_to_avg() you need to:
- Pass the average through forward propagation
- Compute the cost
- Backpropagate to update the softmax parameters
■ Model
# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: model
def model(X, Y, word_to_vec_map, learning_rate = 0.01, num_iterations = 400):
"""
Model to train word vector representations in numpy.
Arguments:
X -- input data, numpy array of sentences as strings, of shape (m, 1)
Y -- labels, numpy array of integers between 0 and 7, numpy-array of shape (m, 1)
word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation
learning_rate -- learning_rate for the stochastic gradient descent algorithm
num_iterations -- number of iterations
Returns:
pred -- vector of predictions, numpy-array of shape (m, 1)
W -- weight matrix of the softmax layer, of shape (n_y, n_h)
b -- bias of the softmax layer, of shape (n_y,)
"""
# Get a valid word contained in the word_to_vec_map
any_word = list(word_to_vec_map.keys())[0]
# Initialize cost. It is needed during grading
cost = 0
# Define number of training examples
m = Y.shape[0] # number of training examples
n_y = len(np.unique(Y)) # number of classes
n_h = word_to_vec_map[any_word].shape[0] # dimensions of the GloVe vectors
# Initialize parameters using Xavier initialization
W = np.random.randn(n_y, n_h) / np.sqrt(n_h)
b = np.zeros((n_y,))
# Convert Y to Y_onehot with n_y classes
Y_oh = convert_to_one_hot(Y, C = n_y)
# Optimization loop
for t in range(num_iterations): # Loop over the number of iterations
for i in range(m): # Loop over the training examples
### START CODE HERE ### (≈ 4 lines of code)
# Average the word vectors of the words from the i'th training example
avg = sentence_to_avg(X[i], word_to_vec_map)
# Forward propagate the avg through the softmax layer.
# You can use np.dot() to perform the multiplication.
z = np.dot(W, avg) + b
a = softmax(z)
# Compute cost using the i'th training label's one hot representation and "A" (the output of the softmax)
cost = -np.sum(np.multiply(Y_oh[i], np.log(a)))
### END CODE HERE ###
# Compute gradients
dz = a - Y_oh[i]
dW = np.dot(dz.reshape(n_y,1), avg.reshape(1, n_h))
db = dz
# Update parameters with Stochastic Gradient Descent
W = W - learning_rate * dW
b = b - learning_rate * db
if t % 100 == 0:
print("Epoch: " + str(t) + " --- cost = " + str(cost))
pred = predict(X, Y, W, b, word_to_vec_map) #predict is defined in emo_utils.py
return pred, W, b
# UNIT TEST
def model_test(target):
# Create a controlled word to vec map
word_to_vec_map = {'a': [3, 3], 'synonym_of_a': [3, 3], 'a_nw': [2, 4], 'a_s': [3, 2], 'a_n': [3, 4],
'c': [-2, 1], 'c_n': [-2, 2],'c_ne': [-1, 2], 'c_e': [-1, 1], 'c_se': [-1, 0],
'c_s': [-2, 0], 'c_sw': [-3, 0], 'c_w': [-3, 1], 'c_nw': [-3, 2]
}
# Convert lists to np.arrays
for key in word_to_vec_map.keys():
word_to_vec_map[key] = np.array(word_to_vec_map[key])
# Training set. Sentences composed of a_* words will be of class 0 and sentences composed of c_* words will be of class 1
X = np.asarray(['a a_s synonym_of_a a_n c_sw', 'a a_s a_n c_sw', 'a_s a a_n', 'synonym_of_a a a_s a_n c_sw', " a_s a_n",
" a a_s a_n c ", " a_n a c c c_e",
'c c_nw c_n c c_ne', 'c_e c c_se c_s', 'c_nw c a_s c_e c_e', 'c_e a_nw c_sw', 'c_sw c c_ne c_ne'])
Y = np.asarray([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1])
np.random.seed(10)
pred, W, b = model(X, Y, word_to_vec_map, 0.0025, 110)
assert W.shape == (2, 2), "W must be of shape 2 x 2"
assert np.allclose(pred.transpose(), Y), "Model must give a perfect accuracy"
assert np.allclose(b[0], -1 * b[1]), "b should be symmetric in this example"
print("\033[92mAll tests passed!")
model_test(model)
Run the next cell to train your model and learn the softmax parameters (W, b). The training process will take about 5 minutes
np.random.seed(1)
pred, W, b = model(X_train, Y_train, word_to_vec_map)
print(pred)
□ Examining Test Set Performance
print("Training set:")
pred_train = predict(X_train, Y_train, W, b, word_to_vec_map)
print('Test set:')
pred_test = predict(X_test, Y_test, W, b, word_to_vec_map)
X_my_sentences = np.array(["i adore you", "i love you", "funny lol", "lets play with a ball", "food is ready", "not feeling happy"])
Y_my_labels = np.array([[0], [0], [2], [1], [4],[3]])
pred = predict(X_my_sentences, Y_my_labels , W, b, word_to_vec_map)
print_predictions(X_my_sentences, pred)
# START SKIP FOR GRADING
print(Y_test.shape)
print(' '+ label_to_emoji(0)+ ' ' + label_to_emoji(1) + ' ' + label_to_emoji(2)+ ' ' + label_to_emoji(3)+' ' + label_to_emoji(4))
print(pd.crosstab(Y_test, pred_test.reshape(56,), rownames=['Actual'], colnames=['Predicted'], margins=True))
plot_confusion_matrix(Y_test, pred_test)
# END SKIP FOR GRADING
What you should remember:
- Even with a mere 127 training examples, you can get a reasonably good model for Emojifying.
- This is due to the generalization power word vectors gives you.
- Emojify-V1 will perform poorly on sentences such as *"This movie is not good and not enjoyable"*
- It doesn't understand combinations of words.
- It just averages all the words' embedding vectors together, without considering the ordering of words.
반응형
CHAPTER 3. 'Emojifier-V2: Using LSTMs in Keras'
You're going to build an LSTM model that takes word sequences as input! This model will be able to account for word ordering. Emojifier-V2 will continue to use pre-trained word embeddings to represent words. You'll feed word embeddings into an LSTM, and the LSTM will learn to predict the most appropriate emoji.
□ Packages
import numpy as np
import tensorflow
np.random.seed(0)
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, Input, Dropout, LSTM, Activation
from tensorflow.keras.layers import Embedding
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.initializers import glorot_uniform
np.random.seed(1)□ Model Overview
□ Keras and Mini-batching
In this exercise, you want to train Keras using mini-batches. However, most deep learning frameworks require that all sequences in the same mini-batch have the same length.
This is what allows vectorization to work: If you had a 3-word sentence and a 4-word sentence, then the computations needed for them are different (one takes 3 steps of an LSTM, one takes 4 steps) so it's just not possible to do them both at the same time.
Padding Handles Sequences of Varying Length
- The common solution to handling sequences of different length is to use padding. Specifically:
- Set a maximum sequence length
- Pad all sequences to have the same length.
□ The Embedding Layer
In Keras, the embedding matrix is represented as a "layer."
- The embedding matrix maps word indices to embedding vectors.
- The word indices are positive integers.
- The embedding vectors are dense vectors of fixed size.
- A "dense" vector is the opposite of a sparse vector. It means that most of its values are non-zero. As a counter-example, a one-hot encoded vector is not "dense."
- The embedding matrix can be derived in two ways:
- Training a model to derive the embeddings from scratch.
- Using a pretrained embedding.
■ sentences_to_indices
# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: sentences_to_indices
def sentences_to_indices(X, word_to_index, max_len):
"""
Converts an array of sentences (strings) into an array of indices corresponding to words in the sentences.
The output shape should be such that it can be given to `Embedding()` (described in Figure 4).
Arguments:
X -- array of sentences (strings), of shape (m, 1)
word_to_index -- a dictionary containing the each word mapped to its index
max_len -- maximum number of words in a sentence. You can assume every sentence in X is no longer than this.
Returns:
X_indices -- array of indices corresponding to words in the sentences from X, of shape (m, max_len)
"""
m = X.shape[0] # number of training examples
### START CODE HERE ###
# Initialize X_indices as a numpy matrix of zeros and the correct shape (≈ 1 line)
X_indices = np.zeros((m, max_len))
for i in range(m): # loop over training examples
# Convert the ith training sentence in lower case and split is into words. You should get a list of words.
sentence_words = [w.lower() for w in X[i].split()]
# Initialize j to 0
j = 0
# Loop over the words of sentence_words
for w in sentence_words:
# if w exists in the word_to_index dictionary
if w in word_to_index:
# Set the (i,j)th entry of X_indices to the index of the correct word.
X_indices[i, j] = word_to_index[w]
# Increment j to j + 1
j += 1
### END CODE HERE ###
return X_indices
# UNIT TEST
def sentences_to_indices_test(target):
# Create a word_to_index dictionary
word_to_index = {}
for idx, val in enumerate(["i", "like", "learning", "deep", "machine", "love", "smile", '´0.=']):
word_to_index[val] = idx;
max_len = 4
sentences = np.array(["I like deep learning", "deep ´0.= love machine", "machine learning smile"]);
indexes = target(sentences, word_to_index, max_len)
print(indexes)
assert type(indexes) == np.ndarray, "Wrong type. Use np arrays in the function"
assert indexes.shape == (sentences.shape[0], max_len), "Wrong shape of ouput matrix"
assert np.allclose(indexes, [[0, 1, 3, 2],
[3, 7, 5, 4],
[4, 2, 6, 0]]), "Wrong values. Debug with the given examples"
print("\033[92mAll tests passed!")
sentences_to_indices_test(sentences_to_indices)
X1 = np.array(["funny lol", "lets play baseball", "food is ready for you"])
X1_indices = sentences_to_indices(X1, word_to_index, max_len=5)
print("X1 =", X1)
print("X1_indices =\n", X1_indices)
Build Embedding Layer
Now you'll build the Embedding() layer in Keras, using pre-trained word vectors.
- The embedding layer takes as input a list of word indices.
- sentences_to_indices() creates these word indices.
- The embedding layer will return the word embeddings for a sentence.
■ pretrained_embedding_layer
# UNQ_C4 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: pretrained_embedding_layer
def pretrained_embedding_layer(word_to_vec_map, word_to_index):
"""
Creates a Keras Embedding() layer and loads in pre-trained GloVe 50-dimensional vectors.
Arguments:
word_to_vec_map -- dictionary mapping words to their GloVe vector representation.
word_to_index -- dictionary mapping from words to their indices in the vocabulary (400,001 words)
Returns:
embedding_layer -- pretrained layer Keras instance
"""
vocab_size = len(word_to_index) + 1 # adding 1 to fit Keras embedding (requirement)
any_word = list(word_to_vec_map.keys())[0]
emb_dim = word_to_vec_map[any_word].shape[0] # define dimensionality of your GloVe word vectors (= 50)
### START CODE HERE ###
# Step 1
# Initialize the embedding matrix as a numpy array of zeros.
# See instructions above to choose the correct shape.
emb_matrix = np.zeros((vocab_size, emb_dim))
# Step 2
# Set each row "idx" of the embedding matrix to be
# the word vector representation of the idx'th word of the vocabulary
for word, idx in word_to_index.items():
emb_matrix[idx, :] = word_to_vec_map[word]
# Step 3
# Define Keras embedding layer with the correct input and output sizes
# Make it non-trainable.
embedding_layer = Embedding(vocab_size, emb_dim, trainable=False)
### END CODE HERE ###
# Step 4 (already done for you; please do not modify)
# Build the embedding layer, it is required before setting the weights of the embedding layer.
embedding_layer.build((None,)) # Do not modify the "None". This line of code is complete as-is.
# Set the weights of the embedding layer to the embedding matrix. Your layer is now pretrained.
embedding_layer.set_weights([emb_matrix])
return embedding_layer
# UNIT TEST
def pretrained_embedding_layer_test(target):
# Create a controlled word to vec map
word_to_vec_map = {'a': [3, 3], 'synonym_of_a': [3, 3], 'a_nw': [2, 4], 'a_s': [3, 2], 'a_n': [3, 4],
'c': [-2, 1], 'c_n': [-2, 2],'c_ne': [-1, 2], 'c_e': [-1, 1], 'c_se': [-1, 0],
'c_s': [-2, 0], 'c_sw': [-3, 0], 'c_w': [-3, 1], 'c_nw': [-3, 2]
}
# Convert lists to np.arrays
for key in word_to_vec_map.keys():
word_to_vec_map[key] = np.array(word_to_vec_map[key])
# Create a word_to_index dictionary
word_to_index = {}
for idx, val in enumerate(list(word_to_vec_map.keys())):
word_to_index[val] = idx;
np.random.seed(1)
embedding_layer = target(word_to_vec_map, word_to_index)
assert type(embedding_layer) == Embedding, "Wrong type"
assert embedding_layer.input_dim == len(list(word_to_vec_map.keys())) + 1, "Wrong input shape"
assert embedding_layer.output_dim == len(word_to_vec_map['a']), "Wrong output shape"
assert np.allclose(embedding_layer.get_weights(),
[[[ 3, 3], [ 3, 3], [ 2, 4], [ 3, 2], [ 3, 4],
[-2, 1], [-2, 2], [-1, 2], [-1, 1], [-1, 0],
[-2, 0], [-3, 0], [-3, 1], [-3, 2], [ 0, 0]]]), "Wrong vaulues"
print("\033[92mAll tests passed!")
pretrained_embedding_layer_test(pretrained_embedding_layer)□ Building the Emojifier-V2
# UNQ_C5 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: Emojify_V2
def Emojify_V2(input_shape, word_to_vec_map, word_to_index):
"""
Function creating the Emojify-v2 model's graph.
Arguments:
input_shape -- shape of the input, usually (max_len,)
word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation
word_to_index -- dictionary mapping from words to their indices in the vocabulary (400,001 words)
Returns:
model -- a model instance in Keras
"""
### START CODE HERE ###
# Define sentence_indices as the input of the graph.
# It should be of shape input_shape and dtype 'int32' (as it contains indices, which are integers).
sentence_indices = Input(input_shape, dtype='int32')
# Create the embedding layer pretrained with GloVe Vectors (≈1 line)
embedding_layer = pretrained_embedding_layer(word_to_vec_map, word_to_index)
# Propagate sentence_indices through your embedding layer
# (See additional hints in the instructions).
embeddings = embedding_layer(sentence_indices)
# Propagate the embeddings through an LSTM layer with 128-dimensional hidden state
# The returned output should be a batch of sequences.
X = LSTM(128, return_sequences = True)(embeddings)
# Add dropout with a probability of 0.5
X = Dropout(0.5)(X)
# Propagate X trough another LSTM layer with 128-dimensional hidden state
# The returned output should be a single hidden state, not a batch of sequences.
X = LSTM(128, return_sequences = False)(X)
# Add dropout with a probability of 0.5
X = Dropout(0.5)(X)
# Propagate X through a Dense layer with 5 units
X = Dense(5)(X)
# Add a softmax activation
X = Activation('softmax')(X)
# Create Model instance which converts sentence_indices into X.
model = Model(inputs=sentence_indices, outputs=X)
### END CODE HERE ###
return model
# UNIT TEST
def Emojify_V2_test(target):
# Create a controlled word to vec map
word_to_vec_map = {'a': [3, 3], 'synonym_of_a': [3, 3], 'a_nw': [2, 4], 'a_s': [3, 2], 'a_n': [3, 4],
'c': [-2, 1], 'c_n': [-2, 2],'c_ne': [-1, 2], 'c_e': [-1, 1], 'c_se': [-1, 0],
'c_s': [-2, 0], 'c_sw': [-3, 0], 'c_w': [-3, 1], 'c_nw': [-3, 2]
}
# Convert lists to np.arrays
for key in word_to_vec_map.keys():
word_to_vec_map[key] = np.array(word_to_vec_map[key])
# Create a word_to_index dictionary
word_to_index = {}
for idx, val in enumerate(list(word_to_vec_map.keys())):
word_to_index[val] = idx;
maxLen = 4
model = target((maxLen,), word_to_vec_map, word_to_index)
expectedModel = [['InputLayer', [(None, 4)], 0], ['Embedding', (None, 4, 2), 30], ['LSTM', (None, 4, 128), 67072, (None, 4, 2), 'tanh', True], ['Dropout', (None, 4, 128), 0, 0.5], ['LSTM', (None, 128), 131584, (None, 4, 128), 'tanh', False], ['Dropout', (None, 128), 0, 0.5], ['Dense', (None, 5), 645, 'linear'], ['Activation', (None, 5), 0]]
comparator(summary(model), expectedModel)
Emojify_V2_test(Emojify_V2)Run the following cell to create your model and check its summary.
- Because all sentences in the dataset are less than 10 words, max_len = 10 was chosen.
- You should see that your architecture uses 20,223,927 parameters, of which 20,000,050 (the word embeddings) are non-trainable, with the remaining 223,877 being trainable.
- Because your vocabulary size has 400,001 words (with valid indices from 0 to 400,000) there are 400,001*50 = 20,000,050 non-trainable parameters.
model = Emojify_V2((maxLen,), word_to_vec_map, word_to_index)
model.summary()
■ Compile the Model
As usual, after creating your model in Keras, you need to compile it and define what loss, optimizer and metrics you want to use. Compile your model using categorical_crossentropy loss, adam optimizer and ['accuracy'] metrics:
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])□ Train the Model
It's time to train your model! Your Emojifier-V2 model takes as input an array of shape (m, max_len) and outputs probability vectors of shape (m, number of classes). Thus, you have to convert X_train (array of sentences as strings) to X_train_indices (array of sentences as list of word indices), and Y_train (labels as indices) to Y_train_oh (labels as one-hot vectors).
X_train_indices = sentences_to_indices(X_train, word_to_index, maxLen)
Y_train_oh = convert_to_one_hot(Y_train, C = 5)
model.fit(X_train_indices, Y_train_oh, epochs = 50, batch_size = 32, shuffle=True)
■ LSTM Version Accounts for Word Order
- The Emojify-V1 model did not "not feeling happy" correctly, but your implementation of Emojify-V2 got it right!
- If it didn't, be aware that Keras' outputs are slightly random each time, so this is probably why.
- The current model still isn't very robust at understanding negation (such as "not happy")
- This is because the training set is small and doesn't have a lot of examples of negation.
- If the training set were larger, the LSTM model would be much better than the Emojify-V1 model at understanding more complex sentences.
■ Congratulations!
You've completed this notebook, and harnessed the power of LSTMs to make your words more emotive! ❤️❤️❤️
By now, you've:
- Created an embedding matrix
- Observed how negative sampling learns word vectors more efficiently than other methods
- Experienced the advantages and disadvantages of the GloVe algorithm
- And built a sentiment classifier using word embeddings!
Cool! (or Emojified: 😎😎😎 )
■ What you should remember:
- If you have an NLP task where the training set is small, using word embeddings can help your algorithm significantly.
- Word embeddings allow your model to work on words in the test set that may not even appear in the training set.
- Training sequence models in Keras (and in most other deep learning frameworks) requires a few important details:
- To use mini-batches, the sequences need to be padded so that all the examples in a mini-batch have the same length.
- An Embedding() layer can be initialized with pretrained values.
- These values can be either fixed or trained further on your dataset.
- If however your labeled dataset is small, it's usually not worth trying to train a large pre-trained set of embeddings.
- LSTM() has a flag called return_sequences to decide if you would like to return every hidden states or only the last one.
- You can use Dropout() right after LSTM() to regularize your network.
■ 마무리
"Sequence Models" (Andrew Ng)의 2주차 "Emojify"의 실습에 대해서 정리해봤습니다.
그럼 오늘 하루도 즐거운 나날 되길 기도하겠습니다
좋아요와 댓글 부탁드립니다 :)
감사합니다.
반응형
'COURSERA' 카테고리의 다른 글
| week 3_Sequence Models & Attention Mechanism 연습 문제 (Andrew Ng) (0) | 2022.05.17 |
|---|---|
| week 3_Sequence Models & Attention Mechanism (Andrew Ng) (1) | 2022.05.10 |
| week 2_Operations on Word Vectors 실습 (Andrew Ng) (0) | 2022.04.27 |
| week 2_Natural Language Processing & Word Embeddings 연습 문제 (Andrew Ng) (0) | 2022.04.27 |
| week 2_NLP and Word Embeddings (Andrew Ng) (0) | 2022.04.27 |
댓글