Introduction

• In classification, it is not necessary to estimate the class densities $p(x|C_i)$ or the exact posterior probability $P(C_i|x)$. we only need to estimate where the class boundaries lie.
• After training, the weight vector can be written down in terms of a subset of the training set, which are the so-called support vectors.
• The output is written as a sum of the influences of support vectors and these are given by kernel functions that are application-specific measures of similarity between data instances.
• No need to represent instances as vectors: Kernel functions can calculate the similarity without needing to represent instances explicitly as vectors.
• Kernel-based algorithms are formulated as convex optimization problems and there is a single optimum.

Optimal Separating Hyperplane

• $r^t(w^Tx^t+w_0) \geq +1$
• We do not simply require $r^t(w^Tx^t+w_0) \geq 0$
• We want the instances some distance away for better generalization.

Margin

• Distance from the discriminant to the closest instances on either side

• Distance of x to the hyperplane: $|w^Tx^t+w_0|/||w||$

  → $r^t(w^x+w_0)/||w|| \geq \rho, \forall t$ ($\rho$ is margin, 2-classes case)

  → infinity solutions

• For a unique solution, fix $\rho ||w||=1$ and to maximize margin, we minimize $||w||$

  $\text{find } ||w|| \text{ to } \min \frac{1}{2} ||w||^2 \text{ subject to } r^t(w^Tx^t+w_0) \geq +1, \forall t$

  

  Circled instances are the support vectors

• Use Lagrange function:

  $L_p = \frac{1}{2}||w||^2 - \sum^N_{t=1} \alpha^t[r^t(w^Tx^t+w_0)-1] \text{ with respect to }\alpha$

  $L_p = -\frac{1}{2}\sum_t\sum_s \alpha^t \alpha^s r^t r^s (x^t)^Tx^s + \sum_t \alpha^t \text{ subject to } \sum_t \alpha^t r^t =0 \text{ and }\alpha^t \geq 0, \forall t$

  • maximize $\alpha$
  • only need the similarity $(x^t)^Tx^s$
  • Most $\alpha^t$ are 0 and only a small number have $\alpha^t>0$; they have the support vector

Soft Margin Hyperplane

• Not linearly separable

  $r^t(w^Tx^t+w_0) \geq 1 - \xi^t$ for slack variable $\xi$

  • If $0 \leq \xi < 1$, $x^t$ is correctly classified but in the margin
  • If $\xi \geq 1$, $x^t$ is misclassified

• Soft error : #{$\xi>0$}

  $\sum_t \xi^t$

  • hard error : #{$\xi \geq 1$}

• New primal is

  $L_p = \frac{1}{2}||w||^2 + C\sum_t\xi^t - \sum_t\alpha^t[r^t(w^Tx^t+w_0)-1+\xi^t]-\sum_t\mu^t\xi^t$

  • maximize $\alpha$, minimize $w$
  • $C$ is the penalty factor as in any regularization scheme trading off complexity (hyperparameter)
  • $C \uparrow$ → strongly penalty soft error → fit training data → increasing variance

(a) $r^tg(x^t)>1, \xi^t=0, \alpha^t = 0$

(b) it is on the right side and on the margin

$\xi^t = 0, 0 < \alpha^t < C$

(c) it is on the right side but is in the margin

$0 \leq \xi^t < 1$, $\xi^t = 1-g(x^t), \alpha^t = C$

(d) it is on the wrong side: misclassification

$\xi^t= 1+g(x^t) \geq 1, \alpha^t = C$