Introduction

• Divide the input space into local regions and learn simple (constant/linear) models in each patch
• Unsupervised: Competitive, Online Clustering
• Supervised: Radial-basis functions, mixture of experts

Online Clustering (Unsupervised)

Online K-Means

• Batch K-Means: $m_i=\frac{\sum_tb_i^tx^t}{\sum_tb_i^t}$
• Online K-Means: data가 순차적으로 들어옴에 따라 $m_i$를 업데이트

Adaptive Resonance Theory (ART)

incremental : one starts with a single group and adds new groups as they are needed

• $b_i = ||m_i-x^t||=\min_{l=1}^k||m_l-x^t||$
• $\begin{cases} m_{k+1} \leftarrow x^t &\text{if }b_i>\rho \\ \Delta m_i = \eta(x^t-m_i) &\text{otherwise} \end{cases}$
  • $\rho$: vigilance (as threshold)

Self-Organizing Maps (SOM)

One way to avoid having dead units is by updating not only the winner but also some of the other units as well.

• When $m_i$ is the closest center, its neighbors are also updated

• ex) the neighborhood size = 2

  → $m_{i-2}, m_{i-1}, m_i, m_{i+1}, m_{i+2}$ are also updated

• In the SOM, not only the closest unit but also its neighbors, in terms of indices, are moved toward the input.

Radial-Basis Function (RBF)

• kinds of activation functions
• sparse representation

Local vs Distributed Representation

Local: 얼마나 먼지 표현 X

Distributed: If use continuous value, not sparse

• Local Representation

  For a given input, only one or a few units are active. It is as if these locally tuned units partition the input space among themselves and are selective to only certain inputs.

• Distributed Representation

  Each hyperplane divides the input space in two, and typically for a given input, many of the hidden units have nonzero output.