Case Study · AI Testing

Validating Book Metadata at Scale with LLMs

An AI-powered book reading and remixing platform needed to know its book metadata was actually correct. Reading every record by hand was impossible, so we used an LLM to do the comparison and turned a slow manual chore into a fast, repeatable check.

Client

AI book reading & remixing platform (anonymised)

Engagement

A focused build, roughly one sprint

Stack

Python, REST API, web scraping, Gemini / GPT

Focus

Data validation & repeatable QA tooling

Background

Why metadata accuracy mattered

When a product relies on accurate information, that information becomes part of the user experience. If the data behind it is wrong, users notice, and trust is hard to win back. That makes verifying it a real testing problem, not an afterthought.

Reputation is on the line

Every time a user opens a book, the metadata around it is part of the impression. Wrong author, wrong category, or a mismatched summary makes the whole product feel unreliable.

Small mistakes, big trust cost

A single incorrect field looks minor on its own, but users lose confidence fast when information they relied on turns out to be inaccurate. The cost of a small error is rarely small.

Accuracy needs verifying

Assuming the data is right is not the same as knowing it. When a product depends on accurate information, you need a reliable, repeatable way to confirm it, especially as the catalogue grows.

The challenge

Too many books to check by hand

The catalogue holds a large and growing number of books, and each one carries several metadata fields that all have to be right. Doing that manually simply does not scale.

Title, author, category, and summary all need to match the source for every book.
Reading and comparing records by hand is slow, repetitive, and error-prone.
Even checking a random subset still takes significant effort each time.
Metadata can change or be reloaded, so yesterday's pass does not guarantee today's.
New books keep getting added, so the check had to be repeatable, not one-off.

The approach

Let an LLM do the comparison

LLMs are good at comparing text and spotting mismatches, so we built a small pipeline around that strength. For each book, the tool gathers two versions of the metadata and asks a model whether they describe the same book.

1. Pull our metadata from the platform API

The tool logs into the platform's API and fetches each book's stored metadata, including a key that identifies which reference record it should map to.

2. Fetch the matching reference record

Using that key, it retrieves the authoritative metadata for the same book from a trusted public reference catalogue, scraping the page details and pulling the summary from the catalogue's API.

3. Compare both with an LLM

Both records go to the model with a focused prompt that checks the fields and returns a clear match or no-match verdict with a short explanation.

4. Review only the mismatches

Instead of reading the whole catalogue, the team only reviews the handful of books the model flags. A runner loops over any range of book IDs, so the check re-runs on demand.

How it works

The pieces under the hood

The whole thing is a few small, single-purpose modules. Each does one job, which keeps the pipeline easy to follow and easy to re-run.

Platform API client

Logs in, fetches a book by ID, and returns its title, author, category, and summary.

Reference fetcher

Looks up the same book in the reference catalogue and returns the trusted fields.

LLM comparator

Sends both records to the model and returns a match verdict with a short reason.

The key idea is the pairing: each book carries an identifier that maps it to the right reference record, so the two metadata sets always line up before the model ever sees them. The prompt itself is deliberately simple and field-oriented:

You are a data validation expert. Compare the following fields
from two datasets and determine if they refer to the same book.

Platform Data:
  Title:    {title}
  Author:   {author}
  Category: {category}
  Summary:  {summary}

Reference Data:
  Title:    {title}
  Author:   {author}
  Category: {category}
  Summary:  {summary}

Is this a match? Respond with a clear "Yes" or "No"
and a brief explanation of why.

Simplified from the real prompt. Asking for a yes / no plus a reason keeps the output easy to scan and easy to act on. We tried more than one model and tuned the wording until the verdicts were reliable for this task.

Results

A manual chore became a repeatable check

The pipeline replaced hours of manual record-reading with a script that surfaces only what actually needs a human's attention.

Hours from idea to a working validation tool

4 fields checked on every book: title, author, category, summary

On demand re-run across any range of book IDs after each reload

Review only the flagged mismatches, instead of the whole catalogue

What we learned

Takeaways from the build

None of this needed deep ML expertise. It needed a clear testing goal and a willingness to treat the LLM as a tool to be checked, not trusted blindly.

Don't blindly trust the output

The model is a helper, not an oracle. We validated its verdicts against known cases before relying on them.

Prompt tuning mattered

Small changes to wording and structure made a real difference. We compared models and picked the one that handled this task best.

You don't need an ML team

A general-purpose LLM and some Python were enough to solve a genuine QA problem end to end.

One idea opens many

Once you see an LLM automate one repetitive testing task, similar opportunities across your QA work start to stand out.