Benchmarking and Stress-Testing Off-the-Shelf Vision Embeddings for Surface Re-Identification in Open-Source Intelligence
- Posted
- Server
- Research Square
- DOI
- 10.21203/rs.3.rs-10189140/v1
Surface re-identification, matching a physical surface such as a wood grain or fabric weave across photographs, is an emerging open-source intelligence (OSINT) technique for linking an anonymous image to a known location. It is increasingly suggested that pretrained vision embeddings can do this off the shelf, but the claim has not been tested against a defined retrieval task with hard negatives and a negative control. We present a reproducible benchmark: retrieving images of the same physical surface from a gallery of many surfaces, we evaluate four off-the-shelf backbone families (CLIP, DINOv2, ResNet-50, EfficientNet-B0) in seven variants as frozen embedders on the KTH-TIPS2-b texture database (4752 images, 44 physical surfaces, with same-material-different-sample images as hard negatives). The strongest model, DINOv2 ViT-L/14, returns the correct surface as its top result 98.1% of the time and verifies same versus different surfaces at AUC 0.973, yet its mean Average Precision is only 0.646 and its equal-error rate is 7.2%, and both degrade across changes of scale and on smooth surfaces. Family matters more than scale: every DINOv2 variant beats every CLIP variant, and the smallest DINOv2 (mAP 0.604) outranks the much larger CLIP ViT-L/14 (mAP 0.590). A nine-by-nine scale-transfer matrix shows retrieval degrading smoothly as the zoom gap widens, and almost every Rank-1 error (98.3%) is a different physical sample of the same material. Repeating the whole benchmark on two further independently captured databases (KTH-TIPS2-a and the original KTH-TIPS) leaves the model ranking and both failure modes unchanged. We conclude that off-the-shelf embeddings are useful as lead generators for a human analyst but not for unqualified identification, and that a self-supervised backbone, not a text-image one, is the better default for surface matching regardless of size.