My own photos overruled the benchmark
The code is on GitHub: Sightline, semantic photo search for Nextcloud.
We used to take photography seriously at home. Now the kids have the cameras, and our Nextcloud server holds about 200,000 photos going back to 2002: raw files from DSLRs, phone shots, screenshots, drone pictures and burst series of the same moment. Folders by date work until you want the photo of the dog in the forest and cannot remember the year.
Google Photos and Apple Photos let you type “dog” and get dogs. Our photos live on our own server, and I wanted the same search there without sending them to a cloud service or keeping a second photo library next to Nextcloud. Over one weekend I built Sightline with an AI coding agent. You type “pies w lesie” (Polish for “dog in the forest”) into the Nextcloud search bar and get photos ranked by what they show.
The decision that mattered most was which model turns photos into something searchable. Published benchmarks and an AI research report both pointed to one model. Four hundred of our own photos, judged blind on the server Sightline runs on, pointed to the other one, and the photos won.
How it finds a dog without tags
Sightline uses a CLIP-family model. It has two encoders trained together: one turns an image into a vector (a list of a few hundred numbers), the other turns text into a vector in the same space. A photo of a dog in a forest lands close to the text “dog in the forest”, and a multilingual model puts “pies w lesie” in the same place. A search encodes the query and returns the nearest photos. Nobody has to tag anything.
When a photo is added or changed, the Sightline app asks Nextcloud’s preview system for a 1024-pixel image and sends it to a small Python service that runs the model on the CPU. Photos and vectors stay on the home server.
The research picked SigLIP 2
Before writing code I had an AI research run go through model licences, published benchmarks and Nextcloud’s source code. It came back with six notes and a report with a clear recommendation: Google’s SigLIP 2 B/16 as the default, and XLM-R B/32, a multilingual CLIP model from LAION, as a fallback in case indexing turned out too slow.
The report got a lot right. It ruled out several of the models that rank best for Polish, because their licences forbid commercial use and every small company running Sightline on its own Nextcloud would inherit that restriction. It found that English-only models lose 40 to 60 points on Polish queries in the multilingual benchmark Immich publishes. In the same table the two candidates were close in Polish: 76.5 for SigLIP 2 and 75.1 for XLM-R. It also said that nobody had published measurements on my server’s CPU, so every performance number would have to come from my own tests.
Stock photos agreed, roughly
The first test ran on my laptop: 678 stock photos and 29 queries, each in English and in Polish. For every query I judged the top 8 results of each model. XLM-R came out slightly ahead at about a third of the encoding cost, and SigLIP 2 lost only 2 points going from English to Polish. Either model would have done.
Our photos did not
Then I ran the same queries on the machine Sightline is built for, a small home server with an Intel N305 processor, using only its CPU. The sample was 400 originals drawn at random from our library: phone photos from 27 different phones, camera JPEGs from 2002 to 2020, screenshots, burst series and 135 raw files. For each query the top 8 of both models were pooled and shuffled, and I marked the relevant photos without knowing which model had found them.
| On our photos, Intel N305 | XLM-R B/32 | SigLIP 2 B/16 |
|---|---|---|
| Relevant photos in the top 8, English queries | 0.73 | 0.77 |
| Relevant photos in the top 8, Polish queries | 0.65 | 0.58 |
| Rank of the first relevant photo (MRR), Polish | 0.80 | 0.64 |
| Image encoding per photo, 8 threads | 50 ms | 190 ms |
The top-8 score is the share of relevant photos among the first eight, adjusted for queries with fewer than eight relevant photos. MRR is 1 divided by the position of the first relevant result, so 0.80 means the first hit is usually at the top.
In English, SigLIP 2 was a little better. In Polish it lost 19 points against its own English score, where on stock photos it had lost 2. XLM-R lost 7. On this CPU SigLIP 2 also costs 3.75 times as much per photo: about 11 hours of encoding for 200,000 photos on all cores, against about 3 hours for XLM-R.
XLM-R became the default. Our family searches in Polish, and the cheaper model was also the better one for those queries.
The numbers have limits. One person judged them, 24 of the 29 queries had a relevant photo in the sample, and each of those had between 1 and 17. Differences of a few points are noise. The Polish gap is the result I trust, because XLM-R was ahead on 11 of the 24 Polish queries and behind on 4.
What a stock set cannot show
Our library had problems a curated set does not have. Raw files are 45 % of it, and Nextcloud core cannot make previews of them. Each raw file carries a JPEG inside it, and that JPEG turned out to be close enough to a full render to search by. None of the embedded JPEGs in the sample had an orientation tag, though, so 16 % of the raw files, the portrait ones, would be indexed lying on their side. Rotated by the orientation stored in the raw file, the embedded JPEG is as similar to the full render as two neighbouring frames of a burst are to each other. Sightline does not index raw files yet; the measurement says how it should.
The search service knows only vectors
One design rule from Sightline is worth copying into any search over private data. The service that holds a vector for every family photo knows only file IDs and vectors, with no users and no Nextcloud credentials. Before a result is shown, Nextcloud checks it against the searching user’s own permissions. A shared photo is indexed once and found by everyone who can open it, and a photo in someone’s private folder never appears for anyone else, whatever the service returns. Who may see a photo is decided in one place, the system that already decides it for files.
Choosing a model on your own data
The method carries over to a company choosing an AI model, and it is cheap. Leaderboards and AI research are good at shortening the list: here they removed models with the wrong licence and models that fail in Polish, and they named what had to be measured. The choice between the last two candidates came from a few hundred items of our own data, judged blind by the person who knows them, on the hardware the system will run on.
Sightline runs on our home server with Nextcloud 35 and is not in the Nextcloud app store yet. The code, the measurements and the architecture notes are on GitHub under the AGPL.