Attention: feature selection

• Attention: feature selection
  • Spatial Attention
  • Channel Attention
• too many features → worse performance: “curse of dimensionality”
• selecting important features → great performance

Spatial Attention

Channel Attention

Squeeze-and-Excitation Net (SENet)

• SE Block
• $F_{sq}$ Squeeze(압축): H×W×C feature map → 1×1×C
  • Global Average Pooling (GAP)
• $F_{ex}$ Excitation(재조정): 1×1×C 벡터로 채널 별 가중치 부여
• Bottleneck 역할
• MobileNet 에서 이용됨

Attention: feature aggregation

• Scaled dot-product attention (Transformers)
  • Relation to GNNs
• Convolution (Relative Positional Embedding)

(Scaled) dot-product attention (Transformer)

• 자신의 점수가 궁금한 Query가 자신과 비슷한 Key들의 Value를 통해 점수를 추측하는 과정이 곧 Attention value를 찾는 것 (출처)
• $\text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^T}{\sqrt{d_k}})V$: Attention Matrix

  • $Q$: Query, $K$: Key, $V$: Value

  • $QK^T = [\text{ sim}_{ij}]$: inner product → similarity between query and key

    • $Q,K$ have row vectors: → $QK^T$: inner product

  • $1/\sqrt{d_k}$ → normalization: prevent Gradient Vanishing (곱이 너무 커지는 것을 방지)

    

General Attention Layer

Outputs:

• context vector: $\bold c$ (shape: D)

Operations:

• Alignment: $e_i=f_{att}(h, x_i)$
  • e.g. $e_i=h\cdot x_i / \sqrt D$ : inner product
  • Distance between data and query
• Attention: $\bold a=\text{softmax}(\bold{e})$
• Output: $\bold c=\sum_i a_i x_i$: weighted average

Inputs:

• Input vectors: $\bold x$ (shape: N×D)
• Query: $\bold h$ (shape: D)
• Attention operation is permutation invariant
  • Doesn’t care about ordering of the features (순서가 바뀌어도 계산 결과는 같음)
  • Stretch H×W=N into N vectors