Skip to content
← Back to blog

Omniscrape: Extract Metadata from Any HTML

Omniscrape is a TypeScript library that reads author and page metadata from HTML: meta tags, JSON-LD, and Next.js or Nuxt hydration data. One call, one result.

By Shaykh Farhan

6 min readWeb Scraping
An author portrait emerging from layers of fragmented information.

Publisher pages do not store an author in one place. One page puts the author in JSON-LD. The next hides the same person in a Next.js hydration payload. Another keeps them in a Nuxt state blob, a meta tag, or a byline in the DOM. If you build scrapers for a living, you know the pattern: you write an extractor for one site, the next site breaks it, and the one parser you meant to write becomes five parsers with five bugs.

I built Omniscrape to end that. It is a TypeScript library that takes raw HTML and a page URL and returns the author, the page metadata, and the evidence behind every field. One call, one result, no browser.

A publisher author page with five numbered hiding places for author data: JSON-LD, Next.js hydration data, Nuxt hydration data, meta tags, and the DOM byline.
One author, five hiding places. Omniscrape reads all of them.

One call

npm install @sfsajid91/omniscrape
import { parseHtml } from "@sfsajid91/omniscrape";

const result = parseHtml({ html, pageUrl: url });

result.author.name;  // "Alice Reporter"
result.author.email; // "writer@example.org"
result.authors;      // resolved author or confirmed co-authors
result.articles;     // discovered articles
result.metadata;     // page title, description, canonical URL
result.provenance;   // evidence behind accepted fields
result.review;       // unresolved identities and conflicting values
result.issues;       // malformed or unsupported sources

That is the whole API surface for most uses: one synchronous function. It does not fetch the page. You fetch it with your own client and pass the final URL, so redirects, retries, and rate limits stay your decision:

const response = await fetch("https://publisher.example/authors/alice");
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const result = parseHtml({ html: await response.text(), pageUrl: response.url });

What it reads

Every extractor runs over the same HTML, and the results are linked into one answer instead of five separate guesses.

Omniscrape pipeline diagram: raw HTML and page URL go into three extractors (DOM and meta tags, JSON-LD graphs, hydration payloads), then a resolution step merges identities on profile URL or publisher ID and sends conflicts to review, producing author, articles, metadata, provenance, review, and issues.
Three extractors, one resolution step, one answer.
Source Support
Next.js __NEXT_DATA__ and recognized inline App Router Flight records
Nuxt Nuxt 3 __NUXT_DATA__ and restricted static Nuxt 2 window.__NUXT__ payloads
Quintype / Bold Recognized static page, story, author, and editorial payloads
Fenrir fenrir-client-data and static fenrirClientData declarations
Generic state Static __INITIAL_STATE__ / __PRELOADED_STATE__ assignments
JSON-LD ProfilePage, Article authors, cross-script @id references, article lists
DOM / meta Scoped author/byline details, contacts, page metadata, article headline links

Hydration payloads can contain author records that are richer than the visible DOM. Omniscrape reads recognized payloads alongside JSON-LD, meta tags, and page content, then resolves the evidence together.

Names lie, profile URLs do not

The hard part of author extraction is not finding a name. It is knowing which name belongs to which person. Two authors can share a name. One author can appear under two spellings. A page can carry the article author, the section editor, and a generic site byline at the same time, and a naive extractor will happily merge all three into one person.

Omniscrape merges author profiles only on an explicit profile URL or a publisher-namespaced ID, never on a matching name alone. Equal slugs cannot override distinct profile URLs or CMS IDs. Article authors stay strictly separate from editorial owners, related story authors, session users, and generic site bylines.

When two sources disagree, it does not guess. If two sources give different email addresses for the same author, both addresses stay in author.emails, author.email is withheld, and the conflict is recorded in review with the evidence for each side:

// Two sources disagree on the author's email.
// Omniscrape keeps both and withholds the single value:
result.author.email;  // withheld: the sources disagree
result.author.emails; // ["alice@publisher.example", "alice@personal.example"]
result.review;        // the conflict, with evidence for both addresses

That is a deliberate design choice. A wrong email in an enrichment pipeline costs more than a missing one. The review bucket exists so a human, or a later pipeline step, can decide with the evidence attached instead of trusting a guess.

Accepted fields carry provenance, including the source URL, DOM or JSON path, scope, and confidence value. If a result looks wrong, that evidence helps trace it back to its source.

Why it never opens a browser

Omniscrape is pure static analysis. No network requests, no browser sandbox, no page script execution. It runs as in-memory compute on the string you hand it and returns a structured object. No side effects, no disk, no I/O.

There is no browser to launch or keep alive, and the work has configurable limits. By default, HTML over 8 MiB is refused, a script over 2 MiB is skipped and recorded, and decoding is bounded by 50,000 nodes and a depth of 128. A broken script records an issue and parsing continues with the remaining sources. Invalid URLs throw TypeError; oversized HTML throws RangeError. Callers remain responsible for network timeouts and transfer limits.

Omniscrape vs Metascraper

Metascraper is the standard general-purpose metadata extractor, and it is good at what it does: titles, descriptions, and Open Graph data across JSON-LD, microdata, meta tags, and DOM bylines. If you need a title and a description, it is the simpler tool.

Omniscrape is narrower and deeper. It targets decoding framework hydration payloads, linking disparate sources into one identity, and determining person-level attribution with explicit conflict handling and provenance trails. If you need to know who wrote the article, and be able to prove it, that is what Omniscrape is for.

Where it fits

Anywhere a pipeline turns publisher pages into structured data: author directories, newsroom enrichment, lead lists built from bylines, and content catalogs that need reliable attribution. It is the author-extraction step in my own scraping work, and it is the reason I stopped writing one-off extractors per site.

Omniscrape is open source under MIT. The package is @sfsajid91/omniscrape on npm, the source is on GitHub, and the case study has the full story of why it exists. Tests run with bun test.