cs224n naive softmax 的推导与实现

Posted on

cs224n-naive-softmax-的推导与实现

[TOC]

0、简介

在cs224n（2019）第二次课后作业Assignment 2的手写作业b（推导梯度公式）和编程作业a中对naive softmax的实现。首先手写作业的推导公式，编程作业则是对这些公式的简单实现

1、公式推出

首先看下背景和问题

image-20190909170321836

image-20190909170250701

首先问题a中已经证明了 损失函数 $ J = CrossEntropy(y, \hat{y})$，而且有$\hat{y}=softmax(U^Tv_c)$, 不妨用$\theta=U^tv_c$,现在有$\hat{y}=softmax(\theta)$, 然后用chain rule推导J关于vc的偏导数。

$$ \J=-y\log{\hat{y}};\ \frac{\partial J}{\partial \hat{y}}=-\frac{y}{\hat{y}} \ \hat{y}=softmax(\theta) \ \text{关于softmax的求导如下：} \ \because y=\frac{e^{x_i}}{\Sigma{e^{xj}}} \ \frac{\partial{y}}{\partial{x_i}} = \frac{e^{x_i}(\Sigma-e^{x_i})}{\Sigma^2}=y(1-y) \ \text{由以上softmax的导数可推：} \ \frac{\partial{\hat{y}}}{\partial{\theta}}=\hat{y}(1-\hat{y}) \ \frac{\partial{\theta}}{\partial{v_c}}=U^T \ \text{由链式法则：} \ \because \begin{aligned} \frac{\partial J}{\partial v_c} &= \frac{\partial J}{\partial \hat{y}} \frac{\partial{\hat{y}}}{\partial{\theta}} \frac{\partial \theta}{\partial v_c} =-y\frac{1}{\hat{y}}\hat{y}(1-\hat{y})U^T=-y(1-\hat{y})U^T \ \end{aligned}$$

以上推导完毕。

2、代码实现

大部分代码是作业自带的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
def naiveSoftmaxLossAndGradient(
    centerWordVec,
    outsideWordIdx,
    outsideVectors,
    dataset
):
    """ Naive Softmax loss & gradient function for word2vec models

    Implement the naive softmax loss and gradients between a center word's 
    embedding and an outside word's embedding. This will be the building block
    for our word2vec models.

    Arguments:
    centerWordVec -- numpy ndarray, center word's embedding
                    (v_c in the pdf handout)
    outsideWordIdx -- integer, the index of the outside word
                    (o of u_o in the pdf handout)
    outsideVectors -- outside vectors (rows of matrix) for all words in vocab
                      (U in the pdf handout)
    dataset -- needed for negative sampling, unused here.

    Return:
    loss -- naive softmax loss
    gradCenterVec -- the gradient with respect to the center word vector
                     (dJ / dv_c in the pdf handout)
    gradOutsideVecs -- the gradient with respect to all the outside word vectors
                    (dJ / dU)
    """

    ### YOUR CODE HERE

    ### Please use the provided softmax function (imported earlier in this file)
    ### This numerically stable implementation helps you avoid issues pertaining
    ### to integer overflow.
    theta=np.dot(centerWordVec, outsideVectors.T)
    y_hat = softmax(theta)

    loss = -np.log(y_hat)[outsideWordIdx] # J = -log(\hat{y_o})
    yhatcopy=y_hat.copy()
    temp=yhatcopy[outsideWordIdx]-1 # -y(1-\hat{y})
    gradCenterVec=np.dot(temp,outsideVectors)
    gradOutsideVecs = np.dot(temp[:, np.newaxis], centerWordVec[np.newaxis,:])

    ### END YOUR CODE

    return loss, gradCenterVec, gradOutsideVecs