CLIP — zero-shot classification

Background

CLIP learns a shared embedding space for images and text. To classify without retraining, embed the image and several candidate captions, then pick the caption whose embedding is most cosine-similar to the image. Scores come from a softmax over the (temperature-scaled) cosine similarities:

$$s_k = \frac{\langle \hat{\mathbf{i}}, \hat{\mathbf{t}}_k\rangle}{\,}\cdot\tau, \qquad p = \text{softmax}(s), \qquad \hat{y} = \arg\max_k p_k$$

where $\hat{\mathbf{i}}, \hat{\mathbf{t}}_k$ are unit-normalized embeddings and $\tau$ is the logit scale.

Problem statement

Implement zero_shot_classify(image_emb, text_embs, temperature=100.0) returning (label, probs).

Input

• image_emb — array-like of length D, the image embedding.
• text_embs — array-like of shape (K, D), one embedding per candidate caption.
• temperature — float, the logit scale (default 100.0).

Output

Returns (label, probs): label is the int index of the best caption; probs is an np.ndarray of K softmax probabilities summing to 1.

Examples

Example 1

Input:  image_emb = [1, 0], text_embs = [[1, 0], [0, 1]]
Output: label = 0, probs ≈ [1.0, 0.0]

Explanation: the image matches caption 0 exactly (cosine 1), caption 1 not at all (cosine 0); with τ=100 the softmax is essentially one-hot.

Constraints

• Unit-normalize the image and each text embedding before the dot product (cosine similarity).
• Scale similarities by temperature, softmax, then argmax.
• Return (int label, np.ndarray probs) with probs summing to 1.

Notes

• This is why CLIP does "open-vocabulary" classification: swap the candidate captions and you've changed the label set with zero training.