Generative Models

• Tractable Density (Tractable: 다루기 쉬운)
  • PixelRNN/CNN
  • Normalizing Flows: RealNVP, Glow
• Approximate density
  • VAE
  • Diffusion Models
• Implicit density
  • GAN

Fully visible belief network (FVBN)

• $p(x)= p(x_1, x_2,...,x_n)=\prod_{i=1}^np(x_i|x_1,...,x_{i-1})$

  

Recurrent Neural Network (RNN)

vs Feed Forward Network (이전까지 배운 networks)

• 내부적으로 사이클 존재. 내부 상태에 따라 input이 달라짐 (기분에 따라 커피맛이 다르듯이)
• h 안에 과거정보를 담은 후 h를 고려하여 output 계산

(Vanilla) RNN

PixelRNN

• Generate image pixels starting from corner
• Dependency on previous pixels modeled using an RNN (LSTM)
• Drawback: sequential generation is slow in both training and inference

PixelCNN

• Generate image pixels starting from corner
• Dependency on previous pixels modeled using a CNN over context region (masked convolution)
• Training faster than PixelRNN (train in parallel)
• Generation is still slow (sequentially)

PixelRNN and PixelCNN

• Pros:
  • Can explicitly compute likelihood p(x)
  • Easy to optimize
  • Good samples