Skip to main content

Write a PREreview

Benchmarking and Stress-Testing Off-the-Shelf Vision Embeddings for Surface Re-Identification in Open-Source Intelligence

Posted
Server
Research Square
DOI
10.21203/rs.3.rs-10189140/v1

Surface re-identification, matching a physical surface such as a wood grain or fabric weave across photographs, is an emerging open-source intelligence (OSINT) technique for linking an anonymous image to a known location. It is increasingly suggested that pretrained vision embeddings can do this off the shelf, but the claim has not been tested against a defined retrieval task with hard negatives and a negative control. We present a reproducible benchmark: retrieving images of the same physical surface from a gallery of many surfaces, we evaluate four off-the-shelf backbone families (CLIP, DINOv2, ResNet-50, EfficientNet-B0) in seven variants as frozen embedders on the KTH-TIPS2-b texture database (4752 images, 44 physical surfaces, with same-material-different-sample images as hard negatives). The strongest model, DINOv2 ViT-L/14, returns the correct surface as its top result 98.1% of the time and verifies same versus different surfaces at AUC 0.973, yet its mean Average Precision is only 0.646 and its equal-error rate is 7.2%, and both degrade across changes of scale and on smooth surfaces. Family matters more than scale: every DINOv2 variant beats every CLIP variant, and the smallest DINOv2 (mAP 0.604) outranks the much larger CLIP ViT-L/14 (mAP 0.590). A nine-by-nine scale-transfer matrix shows retrieval degrading smoothly as the zoom gap widens, and almost every Rank-1 error (98.3%) is a different physical sample of the same material. Repeating the whole benchmark on two further independently captured databases (KTH-TIPS2-a and the original KTH-TIPS) leaves the model ranking and both failure modes unchanged. We conclude that off-the-shelf embeddings are useful as lead generators for a human analyst but not for unqualified identification, and that a self-supervised backbone, not a text-image one, is the better default for surface matching regardless of size.

You can write a PREreview of Benchmarking and Stress-Testing Off-the-Shelf Vision Embeddings for Surface Re-Identification in Open-Source Intelligence. A PREreview is a review of a preprint and can vary from a few sentences to a lengthy report, similar to a journal-organized peer-review report.

Before you start

We will ask you to log in with your ORCID iD. If you don’t have an iD, you can create one.

What is an ORCID iD?

An ORCID iD is a unique identifier that distinguishes you from everyone with the same or similar name.

Start now