How I look for lost texts — Maciej Janowski

Libraries have put hundreds of thousands of manuscript pages online. Most are catalogued well. The fragments are not. A fragment is usually binding waste: a leaf cut up centuries ago to stiffen a newer book’s cover, found again at restoration, and photographed under a label like “unidentified religious treatise”. That is the gap I work in, from a desk in Japan, with AI reading at scale and me judging. Here is how a pass works, and how it goes wrong.

Harvest, then choose

The first round ran over Fragmentarium, a shared online catalogue of manuscript fragments. It publishes IIIF manifests, machine-readable lists of each fragment’s images and metadata. I harvested 6,738 manifests across 286 collections. By catalogue title, 433 were unknown or unidentified and 182 had only a genre. I chose 2,000, weighted to the unidentified and the vague, plus vernacular pieces, partial identifications and two control groups of already-identified fragments. 1,956 had a usable image.

Each image goes to a vision model with no catalogue text at all: the picture only, as an overview and four zoomed tiles. The model writes a diplomatic reading of the first lines, an expansion, a guess at script and date, the language, a guess at the work with a confidence, and a few phrases it is sure of. A model that has seen “Augustine” in the catalogue will read Augustine; blind, its reading can be checked against the catalogue later.

The readings are working transcriptions, corrected by eye when a candidate comes up for review. They are not editions, not a text to cite.

Matching, and why “no match” is not “unknown”

The readings are searched against a local index of about 2.6 million overlapping passages from open corpora: the Vulgate, the early volumes of the Patrologia Latina and the Vienna corpus (collections of the Latin church writers), the breviary and missal, chant texts. The phrases the transcriber was sure of are then sent, three words at a time, to Corpus Corporum, a large online library of Latin texts. A hit at any stage marks the item as known.

Everything else is flagged “no match”, and that flag is weak. On the validation set, about one in seven legible known texts came back falsely “unknown”, clustered where the open corpora are thin: lexica, medicine, canon law, music theory. Nothing open covers vernacular texts at all. So “no match” means “not in what I can search”. It is a reason to go and look, not a finding.

Rank, then check by hand

A ranker scores each item on legibility, how little matched, and whether the catalogue already had an identification. The hand-checks start at the top: every page of the fragment beside the model’s reading, word by word, then the reference tools and the holding library’s own records.

The validation set was 55 fragments whose catalogue names the work, the answer key. Only 37 were legible at all; binding strips and show-through beat the model as they beat the eye. Of those 37, the pipeline named the right work for 28, about three in four, the genre only for 4, and the rest came back falsely unknown. Of the 24 top candidates I checked by hand, about one in three led to something a specialist or library would want: an identification, a correction, or a find. Six were pipeline artefacts.

What goes wrong

The artefacts have shapes. A worker holding 12 items writes item B’s reading under item A’s id; now a swap check compares each reading’s date with the catalogue’s, and items are written down one at a time. The model reads the host instead of the fragment: a label, a printed leaf of the book it was bound into, the neighbour on the same photograph. These float to the top as strange unidentified texts, so every reading now names its layer. Workers read one side to save cost, and the decisive words kept turning up on the other, so top candidates now get every image. Bible quotations fool the matcher, since every sermon quotes the Bible; one treatise “matched” a commentary on Matthew through a shared verse. And web search cannot reach early printed books: two Romansh hymns with no hit anywhere turned out to be in a 1562 psalter. The rule since: find the genre and model text behind a vernacular piece, then search scanned editions by hand.

Then a second pair of eyes: before anything goes out, a second model gets the verdict and evidence with one instruction, break it. In the first round that caught the printed source of those hymns and two misreadings. Only then do I write to a specialist or the holding library and ask.

The second round

The second round ran over the Handschriftenportal, the shared catalogue of manuscripts in German libraries: 481 fragments transcribed blind, about 9.8 million model tokens in all. Three things changed. The top 60 candidates were triaged against the catalogue’s own full descriptions before any hand-check, which cut them to 18, because a generic title often hides a text the cataloguer named lower down. Those 18 were read on every page, which changed the verdict on five. And 13 of the claims went to the second model to break.

No new text survived. There is one unrecognised witness of a known work and about eight catalogue identifications or corrections. About one in two of the hand-checked items gave a library something to fix, against one in three in the first round, but nothing reached the level of the Antwerp fragment: a better-catalogued corpus hides mostly common texts. None of it has been reported yet, and it will reach the findings page only after the libraries have heard.

What the labels say

Every result lives on the findings page with a status that says how far it has been checked, and no further. The Antwerp Vision of Tnugdal is marked confirmed, because Mattia Cavagna at UCLouvain, editor of the French versions, confirmed it. Two Innsbruck fragments, the Romansh hymns and a German exposition of Isaiah, are marked acknowledged: the library said they were new to its records, which is not peer review, and the hymns are not a new text. Leuven Ms. 1317 is acknowledged on a first reply: the curator found it convincing at first sight and will check it further. Two Frankfurt fragments are pending, reported with no answer yet. One weak lead is parked, not a claim. The rest are catalogue corrections, not discoveries.

None of this is a lost work found. The best of it is a known story in a version nobody had recorded. The method is AI-assisted. The hard parts, the judgement and the checking, are not.