Object-Object-Interaction Detection

• Task Definition:
  • Detect objects from an image
  • Predict the interaction between objects
• Goal: Predict triplet < obj1, relation, obj2 >
  • ex. < person, ride, motorcycle >

Challenges in Scene Graph Generation

• complex semantics/concepts

• Large label spaces (triplets)

  < obj1, relation, obj2 > → 100 * 100 * 100 = 1M → huge…

• Limited data

  • label space에비해 data 수가 적음

• Zero-shot Learning: Predict unseen triplets

  • 예를 들어, 기존 데이터에서는 곰이 오토바이를 탄다는 triplet이 없었으나 곰이 오토바이를 타고 있는 사진이 주어지면 이를 맞춰야함.
  • 학습 데이터에서 한 번도 못 본 것을 task에서 맞춰야 하는 것이 zero-shot learning

Relation Tensor

• Relation Tensor: < obj1, relation, obj2 >
• Likelihood of relations given two objects
• Sparse tensor
• Tensor factorization of label space
  • such as matrix factorization
  • similar to PCA (using Eigenvector/value Decomposition)

Scene Graph Generation (Advanced?)

1. Faster RCNN → features of object bounding boxes (nodes) and intersecting bounding boxes (edges)
2. Tensor factorization → Using the detected labels, predict predicates

Human Object Interaction Detection

• Always involve human