Introduction

Zero-shot sketch-based image retrieval (ZS-SBIR) is a central problem to sketch understanding [6]. This paper aims to tackle all problems associated with the current status quo for ZS-SBIR, including category-level (standard) [4], fine-grained [1], and cross-dataset [3]. In particular, we advocate for (i) a single model to tackle all three settings of ZS-SBIR, (ii) ditching the requirement on external knowledge to conduct category transfer, and more importantly, (iii) a way to explain why our model works (or not). Our solution first is a transformer-based cross-modal network, that (i) sources local patches independently in each modality, (ii) establishes patch-to-patch correspondences across two modalities, and (iii) computes matching scores based on putative correspondences. We approach (i) by proposing a novel CNN-based learnable tokenizer, that is specifically tailored to sketch data. This is because the vanilla non-overlapping patch-wise tokenization proposed in ViT [2] is not friendly to the sparse nature of sketches (as most patches would belong to the uninformative blank). In the same spirit of class token developed in ViT for image recognition, we introduce a learnable retrieval token to prioritize tokens for cross-modal matching. To establish (ii) patch-to-patch correspondences, a novel cross-attention module is proposed that operates across sketch-photo modalities. Specifically, we propose cross-modal multi-head attention, in which the query embeddings are exchanged between sketch and photo branches to reason patch-level correspondences with only category-level supervision. Finally, with the putative matches in place, inspired by relation networks [5], we propose a kernel-based relation network to aggregate the correspondences and calculate a similarity score between each sketch-photo pair. We achieve state-of-the-art performance across all said ZS-SBIR settings with some illustration in Figure 1.

Figure 1.  Attentive regions of self-/cross-attention and the learned visual correspondence for tackling unseen cases. (a) The proposed retrieval token [Ret] can attend to informative regions. Different colors are attention maps from different heads. (b) Cross-attention offers explainability by explicitly constructing local visual correspondence. The local matches learned from training data are shareable knowledge, which enables ZS-SBIR to work under diverse settings (inter- / intra-category and cross datasets) with just one model. (c) An input sketch can be transformed into its image by the learned correspondence, i.e., sketch patches are replaced by the closest image patches from the retrieved image.

Our Solution

Technically, our method is built over a transformer-based cross-modal network, with three novel components (i) a self-attention module with a learnable tokenizer to produce visual tokens that correspond to the most informative local regions, (ii) a cross-attention module to compute local correspondences between the visual tokens across two modalities, and finally (iii) a kernel-based relation network to assemble local putative matches and produce an overall similarity metric for a sketch-photo pair. Experiments show ours indeed delivers superior performances across all ZS-SBIR settings. And the important goal of explainability is elegantly achieved by visualizing cross-modal token correspondences, and for the first time, via sketch to photo synthesis by replacement of all matched photo patches in the gallery, as shown in Figure 1(c).

As shown in Figure 2, (a) a learnable tokenizer using filters with large receptive field generates structure-preserving tokens, preventing generation of the uninformative tokens. (b) A self-attention module finds the most informative regions of each input sketch/photo for the subsequent cross-modal local matching. Meanwhile, the original classification token [CLS] is replaced by a retrieval token [RET] for the retrieval task. (c) A cross-attention module learns the visual correspondences between generated visual tokens from each modality. (d) A token-level relation network infers the final matching results based on the local correspondences generated by the cross-attention module.

Figure 2.  Network overview.

Experimental Results

Table 1.  Category-level ZS-SBIR comparison results. “ESI” : External Semantic Information. “-” : not reported. The best and second best scores are color-coded in red and blue.

Table 2.  Zero-shot FG-SBIR results (%). Note that all competitors are not zero-shot models, they are trained on Chair-V2. Best in bold.

Table 3.  Cross-dataset ZS-SBIR results. “S”, “T” and “Q” denote Sketchy Ext, TU-Berlin Ext, and QuickDraw Ext, respectively. “(·)” denotes the number of test categories which are unseen to ensure the zero-shot setting. E.g., S->T(21) denotes that, we train on the training split of Sketchy Ext, then test on a subset (21 unseen classes) of the testing split of TU-Berlin Ext. Rows with a grey background indicate using ViT backbone for fair comparisons. Best in bold.

The experimental findings across various settings, including category-level zero-shot SBIR, zero-shot fine-grained SBIR, and cross-dataset zero-shot SBIR, as presented in Tables 1-3, demonstrate the effectiveness of our proposed approach. Our method consistently outperforms existing state-of-the-art (SoTA) techniques in the majority of cases.

Conclusion

In our endeavor to address Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) and emphasize the importance of interpretability, we drew inspiration from traditional vision techniques – bag-of-words. Our novel approach introduces a patch matching framework that not only achieves explainability but also effectively handles all ZS-SBIR scenarios simultaneously. We firmly believe that our advanced retrieval system will significantly impact the progress of ZS-SBIR and bring SBIR closer to practical applications.

Link to the paper