Newspaper Atlas

What it is

I wanted to find out whether a private search system could connect an 1865 newspaper report and a 2022 headline in another language when the two share no words. I tested it on the material behind The Speed of News: 1,099 passages from American newspapers (1799–1963) and 7,976 headlines in 58 languages about three recent events.

Using the map

Each dot is one text, its position on the map reflects its meaning so that texts about similar things sit close together in any language, and its color shows the era. Selecting a dot displays the text and its eight closest neighbors by meaning, joined by lines because those neighbors can be far apart on the map. You can also pick one of the example questions: the diamond marks the question, the numbered dots are the six texts retrieved for it, and the recorded answer below was written using only those six texts.

What turned up

• For 96% of texts, the nearest neighbor by meaning belongs to the same event, and half of all neighbors are in a different language from the text they neighbor.
• The question "Our beloved general is dead" retrieves reports of General Jackson's death in 1845 and Lincoln's in 1865, along with 2022 headlines in several languages about the death of Queen Elizabeth II.
• A Spanish question about the Notre-Dame fire retrieves Arabic, Spanish, and English headlines, and an English question about rubble and survivors retrieves a Portuguese headline first.
• The older newspaper passages, from 1799 to 1865 and the Kennedy assassination, are crowded into one tight region where their events overlap, probably because the scans are garbled and the model treats much of that noise as similar.
• The question about a peace treaty has one strong match, the Treaty of Ghent, and its other five results come from unrelated events and have low scores.

How it was built

The texts are newspaper passages from the Library of Congress's Chronicling America, as read from scanned pages, and the first headline each news site published about an event, as found by GDELT. I removed bare site names and repeated headlines. These are the same sources used in The Speed of News, and every text links to its original.

A small multilingual embedding model (paraphrase-multilingual-MiniLM-L12-v2) converts each text into 384 numbers, so that texts with similar meaning receive similar numbers. UMAP reduces those numbers to two dimensions to make the map. Neighbors and retrieval both use cosine similarity, and each question is placed on the map at the similarity-weighted average position of the texts retrieved for it.

The recorded answers were written by ministral-3, a language model that runs on my laptop through Ollama. I instructed it to answer using only the six retrieved texts, to cite them, and to say so if they did not contain the answer. The answers are shown exactly as the model wrote them.

I ran everything on my own laptop and sent no text to an outside service. The page loads a static file of precomputed results, so nothing a visitor does leaves their browser. A private search over an organization's own documents follows the same steps: embed the documents, map them, retrieve the closest ones to a question, and answer from those alone.

What to keep in mind

• I chose the example questions to show what the method does well and where it falls short, so they do not measure its accuracy.
• The embedding model is small: it handles common languages well, rarer languages less well, and heavily garbled scans poorly.
• A similarity score is not a probability, and a high score means the wording and topic are similar and does not mean the answer is correct.
• The recorded answers come from a small model and can be wrong, so they should be checked against the sources.
• The headlines belong to their publishers, and I show them briefly with links to the originals.