Figureure 1 · Previous work illustrated an intra-modal misalignment in CLIP space byFigure 1. Previous work illustrated an intra-modal misalignment in CLIP space by showing there are cat images closer to a dog (d̸=) than to another cat (d=). We argue $d _ { \neq } < d _ { = }$ is no sign of misalignment. For a labeled downstream dataset, intra-class variance of open vocabulary models is expected and desired to capture semantics and style beyond the narrow dataset-specific labels. Classifying and retrieving with frozen CLIP image embeddings still works well when similarities are measured along the datasetspecific semantic axes. Here the horizontal axis captures dog/cat.这张图来自论文 PDF 的结构化抽取。当前用于辅助理解 Reevaluating CLIP Intra-Modal Misalignment 的方法或实验，请结合正文精读段落一起看。Figureure 2 · Pairwise cosine similarity distributionsFigure 2. Pairwise cosine similarity distributions. Left: Similarities between same class (blue) and opposite class (orange) image feature pairs. A high overlap ratio between the two colors was previously highlighted as an indicator for an intra-modal misalignment issue in CLIP. Right: Similarity distributions of image-text pairs (purple) versus image-image pairs (green). Because CLIP is only supervised on the former, the divergence has previously prompted concerns about whether the latter reflect true similarities. CLIP ViT-B/16. Dataset as in Tab. 1.这张可视化用来解释 Reevaluating CLIP Intra-Modal Misalignment 学到的中间表征或对齐关系。重点看它是否支持正文里的机制判断。
