### Idea behind word2vec Model

“You should know a word by the company it keeps.”

The Word2Vec technique is based on a feed-forward, fully connected architecture. To give an overview of word2vec algorithm ,let’s train a neural network to do the following. Given a specific word in the middle of a sentence (the input word), look at the words nearby and pick one at random. The network is going to tell us the probability for every word in our vocabulary of being the โnearby wordโ that we chose. When I say “nearby”, there is actually a “window size” parameter to the algorithm. A typical window size might be 5, meaning 5 words behind and 5 words ahead (10 in total). The output probabilities are going to relate to how likely it is find each vocabulary word nearby our input word.

For example, if you gave the trained network the input word โSanโ, the output probabilities are going to be much higher for words like โDiegoโ and โFranciscoโ than for unrelated words like โAppleโ and โOrangeโ.

Weโll train the neural network to do this by feeding it word pairs found in our training documents. The below example shows some of the training samples (word pairs) we would take from the sentence โPM Modi and the American president Trump addressed the Houston rally.โ Iโve used a small window size of 2 just for the example. The word highlighted in blue is the input word.

The network is going to learn the statistics from the number of times each pairing shows up. So, for example, the network is probably going to get many more training samples of (โSanโ, โDiegoโ) & (“San” , “Francisco”) than it is of (โSanโ, โYorkโ). When the training is finished, if you give it the word โSanโ as input, then it will output a much higher probability for โDiegoโ or โFranciscoโ than it will for โYorkโ.

In our example above the network is probably going to get many more training samples of** (โPMโ, โModiโ)** than it is of (**โPMโ, โTrumpโ)**. When the training is finished, if you give it the word โ**Modi**โ as input, then it will output a much higher probability for โ**PM**โ than it will for โpresidentโ or “**Trump**“.

During training, word2vec actually picks a random window size between 1 and window_size. This has the (indirect) effect of giving different weight to the context words based on their distance from the center word. Weโll discuss this further.

**Different types in Word2vec :-**

Word2vec comes in two variants–the skip-gram model and the Continuous Bag-of-Words (CBOW) model-

**Skip-grams (SG):**Predict context words (position independent) given center word.**Continuous Bag of Words (CBOW):**Predict center word from (bag of) context word.

**Skip-gram Model Architecture**

First of all, you know you canโt feed a word just as a text string to a neural network, so we need a way to represent the words to the network. To do this, we first build a vocabulary of words from our training documentsโletโs say we have a vocabulary of 10,000 unique words. Weโre going to represent an input word like โantsโ as a one-hot vector. This vector will have 10,000 components (one for every word in our vocabulary) and weโll place a โ1โ in the position corresponding to the word โantsโ, and 0s in all of the other positions. The output of the network is a single vector (also with 10,000 components) containing, for every word in our vocabulary, the probability that a randomly selected nearby word is that vocabulary word. Hereโs the architecture of our neural network.

There
is **no activation function on the hidden layer
neurons**, but the **output neurons use
softmax**. Kindly note that **when training** this network on word pairs, the input is a one-hot vector representing the
input word and the **training output is also a
one-hot vector representing the output word.** But when you **evaluate the trained network on an input word**,
the **output vector will actually be a probability
distribution** (i.e., a bunch of floating point values, not a one-hot
vector).

**The Hidden Layer**

For our example, weโre going to say that weโre learning word vectors with 300 features. So the hidden layer is going to be represented by a weight matrix with 10,000 rows (one for every word in our vocabulary) and 300 columns (one for every hidden neuron).

300 features is what Google used in their published model trained on the Google news dataset (you can download it from here). The number of features is a “hyper parameter” (which is nothing but the Embedding Dimension of each word) that you would just have to tune to your application (that is, try different values and see what yields the best results).

General approach for dealing with words in your text data is to one-hot encode your text. You will have thousands or millions of unique words in your text vocabulary. Computations with such one-hot encoded vectors for these words will be very inefficient because most values in your one-hot vector will be 0. So, the matrix calculation that will happen in between a one-hot vector and a first hidden layer will result in a output that will have mostly 0 values

We use **embeddings **to
solve this problem and greatly improve the efficiency of our network.
Embeddings are just like a fully-connected layer. We will call this layer asโ
embedding layer and the weights as โ embedding weights.

Now, instead of doing the matrix multiplication between the inputs and hidden layer we directly grab the values from embedding weight matrix. We can do this because the multiplication of one-hot vector with weight matrix returns the row of the matrix corresponding to the index of โ1โ input unit

If you look at the rows of this weight matrix, these are actually what will be our word vectors!

If you multiply a 1 x 10,000 one-hot vector by a 10,000 x 300 matrix, it will effectively just select the matrix row corresponding to the โ1โ. Hereโs a small example to give you a visual.

So, we use this Weight Matrix as lookup table. We
encode the words as integers, for example โcoolโ is encoded as 512, โhotโ is
encoded as 764. Then to get hidden layer output value for โcoolโ we just simply
need to lookup the 512th row in the weight matrix. This process is called **Embedding
Lookup**. The number of dimension in the
hidden layer output is the **embedding dimension**

*Kindly note that at the very beginning of training, all weights in the Embedding matrix are initialized to random values.*

Note:-

Quality of word embedding increases with higher dimensionality. However, after reaching some threshold, the marginal gain will diminish. Typically, the dimensionality of the vectors is set to be between 100 and 1,000.

**Output Layer**

The 1 x 300 word vector for โantsโ then gets fed to the output layer.

In
order to guarantee a probability based representation of the output word, a softmax activation function is
used in the output layer and the following error function *E* is adopted during training:

wherewois the output word andwiis the input word. At the same time, to reduce computational effort, a linear activation function is used for the hidden neurons and the same weights are used to embed all inputs (CBOW) or all outputs (Skip-gram).

Since the output layer is a **softmax regression classifier** so each output neuron (one per word in our vocabulary!) will produce an output between 0 and 1, and the sum of all these output values will add up to 1. Specifically, each output neuron has a weight vector which it multiplies against the word vector from the hidden layer, then it applies the function exp(x) to the result. Finally, in order to get the outputs to sum up to 1, we divide this result by the sum of the results from all 10,000 output nodes. Hereโs an illustration of calculating the output of the output neuron for the word โcarโ.

*Kindly note that at the very beginning of training, all weights in the Output Matrix are set to 0.*

**Challenges**

You may have noticed that the skip-gram neural network contains a huge number of weights… For our example with 300 features and a vocab of 10,000 words, that’s 3M weights in the hidden layer and output layer each! Training this on a large dataset would be prohibitive, so the word2vec authors introduced a number of tweaks to make training feasible.

We’ll cover this in our next post. So stay tuned!!

0