1. Add Self-Attention blocks

Papers

Self-Attention Generative Adversarial Networks

Self-Attention Generative Adversarial Networks

Non-local Neural Networks

Non-local Neural Networks

SAGAN

Self-Attention for CNNs

• 각 Self-Attention Input Vectors 로 Channel 별로 나눠서 넣음 ($C\times 1 \rightarrow \# H\times W$ )
• Model is still a CNN! Can we replace convolution entirely?

2. Replace Convolution with “Local Attention”

Papers

Local Relation Networks for Image Recognition

Local Relation Networks for Image Recognition

Stand-Alone Self-Attention in Vision Models

Stand-Alone Self-Attention in Vision Models

Local Attention