A Descriptor for large-scale Sketch-Based Image Retrieval

• Pros:
– translation invariance – local deformation inv.
• Cons: slow
db: 205 oil-paintings
Jacobs et al., 1995
Fast Multiresolution Image Querying
db: 20.000 images
PREVIOUS WORK
Chang et al., 1979
Query-by-Pictorial-Example
• query language for retrieval • features: points/lines/areas • query: point to features of existing images
• 1.5 Mio images – fast query to achieve feature vector interactivity
1.5 Mio feature extraction
0.21 0.13 0.75 0.31 0.41 0.06 0.93 0.17
Comparing sketch and image
Cole et al. – SIGGRAPH 2008
Where do people draw lines?
• People draw lines mostly along
– – strong image gradients
Sketch based image retrieval
S13
0.93 0.12 . . .
Search
0.21 0.13 0.75 0.31 0.41 0.06 0.93 0.17
0.01 0.17 0.68 0.31 0.44 0.05 0.97 0.13
r
feature vectors
Clustering
• Given n result images, “sort” according to color
– perspective invariant matching?
Observations III
• Users often sketch “symbols” instead of shape
– can we match symbols?
Observation IV
• Evaluation is hard
• by words
Communicating images
• by example
/kehinger/class/
Communicating images
• by sketched color areas
Jacobs et al. - SIGGRAPH 1995
Motivation: binary sketches
• feels natural • rather easy to draw • hard facts?
Elder - IJCV 1999
Are Edges incomplete?
original edges
reconstructed
reblurred
• wavelet transform of both sketch and images • descriptor: largest magnitude wavelet
Del Bimbo et al., 1997
Elastic Matching of User Sketches
• User sketch as template • Bend & stretch energy matches Canny edges
Tensor Descriptor
median rank
-100 px
-50 px
0 px
50 px
100 px
-100 px
-50 px
0 px
50 px
100 px
Comparable results for rotated/scaled case
Subset of top 50
Top 15 results
Subset of top 50
Observations I
• Semantic information missing
– make use of existing image tags
Observations II
• Sketches often show side or frontal view
– Clustering using k-means
Sketch-based image retrieval
RESULTS & CONCLUSION
• database stats
1.5 Mio Images
– 1.5 Mio images – 375 GB – 250kb / image
offline feature extraction
– users should rate relevance of a result
query original: rank 5474 top 50 result
Take home message
• SBIR using binary outline sketches
– interactive retrieval results – lots of data allows use of simple descriptors
Communicating images
• by binary outline sketch
• Objective: large scale sketch-based image retrieval
1.5 Mio
Sketch-Based Image Retrieval
WHY BINARY SKETCHES?
A DESCRIPTOR FOR LARGE-SCALE SKETCHBASED IMAGE RETRIEVAL
Mathias Eitz, Kristian Hildebrand, TamyBoubekeur and Marc Alexa
Motivation: image retrieval
Communicating images
H12
H13
H11
0.23 0.44 0.31 0.07 0.93
H12
0.93 0.12 . . .
Tensor Descriptor
Encodes main gradient orientation
structure tensor Sij
S11
feature vector S12 S13 S11 0.21 0.03 0.71 0.32 S12 0.11 0.17 0.93
db: 100 images
Chalechale et al., 2005
Angular Partitioning
db: 3600 images
Sketch based image retrieval
SHAPE DESCRIPTORS
Preconditions
• Binary sketches as input – use gradient orientation
• Sketches/images represented as feature vector
– encode essential image/sketch attributes – small memory footprint – fast to compare
dist
,
0.21 0.13 0.75 0.31 0.41 0.06 0.93 0.17
• offline feature extraction
– 70 images/sec – 8-core Xeon desktop
• search timing
– ~1.5 seconds
Evaluation setup
• Manually defined ground-truth for 27 sketches • What is the rank of the original image?
... ...
– for rotated, translated and scaled sketch?
... ...
Translation invariance
Evaluation
Edge Histogram Descriptor
median rank
100.000 10.000 1000 100 10 1
0.01 0.17 0.68 0.31 0.44 0.05 0.97 0.13
m
Descriptors
• Two types of descriptors
Edge Histogram Descriptor as baseline Tensor descriptor
• Feature extraction from sketch and image equivalent
Hirata et al., 1992
Query by Visual Example
• Search:
– subdivide into 8x8 blocks – compute local correlation between all blocks – sum of all local correlation values gives result
THANKS
Feature extraction
Feature extraction
Feature extraction