Paper

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Show, Attend and Tell: Neural Image Caption Generation with Visual...

Basic Architecture

Model Details

Goal: Single raw image → caption $\mathbf y$ encoded as a sequence of 1-of-K encoded words

• $y=\{\mathbf y_1,...,\mathbf y_C\}, \mathbf y_i\in\R^K$

  K: vocabulary size, C: length of the caption

• Alignment score $e_{t,i,j}=f_\text{att}(s_{t-1}, h_{i,j})$
• Attention weight $\alpha_{t,:,:}=\text{softmax}(e_{t,:,:})$
• Context vector $c_t = \sum_{i,j}a_{t,i,j}h_{i,j}$ → spatial average

Repeat…

Visual Results