Skip to content

Omniscrape

A TypeScript library that reads author and page metadata from HTML you already fetched, including JSON-LD and Next.js or Nuxt hydration data.

Role
Creator
Timeline
2026
Status
Open source
Category
Open source library
Technologies
TypeScript · Bun · Zod
Terminal card showing the parseHtml call from Omniscrape, with author, provenance, and review fields.
Terminal card showing the parseHtml call from Omniscrape, with author, provenance, and review fields.

Architecture & Key Constraints

Authors merge only on profile URLs or publisher IDs, and a conflicting email is kept in review instead of being picked.

What it is for

Publisher pages do not store an author in one place. One page uses JSON-LD. The next hides the same person in a Next.js hydration payload, or in a Nuxt state blob. Omniscrape is the parser I use so that work is not rewritten per site.

parseHtml takes raw HTML and the final page URL. It returns metadata, the author or confirmed co-authors, articles on the page, a provenance trail, and a review bucket. It does not fetch the page. The caller does that.

npm install @sfsajid91/omniscrape

What it reads

It reads JSON-LD, DOM and meta tags, and hydration payloads: Next.js (__NEXT_DATA__ and recognized App Router Flight records), Nuxt 2 and 3, Quintype/Bold, Fenrir, and static __INITIAL_STATE__ or __PRELOADED_STATE__ blobs.

Authors merge only on an explicit profile URL or a publisher id, not on a matching name. Article authors stay separate from editorial owners and site bylines. If two sources disagree on an email, the addresses stay in author.emails, author.email is left empty, and the conflict is recorded in review. A broken script records an issue and parsing continues.

The defaults refuse HTML over 8 MiB, skip a script over 2 MiB, and stop at 50,000 decode nodes or a depth of 128. Those limits are in the function options. Invalid URLs throw TypeError. Oversized HTML throws RangeError.

Unlike general metadata extractors such as Metascraper, Omniscrape is built for framework hydration payloads and person-level attribution, with conflict handling and provenance trails. I wrote about how it works: Omniscrape: Extract Metadata from Any HTML.

import { parseHtml } from "@sfsajid91/omniscrape";

const result = parseHtml({ html, pageUrl: url });
result.author.name;
result.provenance;
result.review;

The package is @sfsajid91/omniscrape on npm (v0.2.2), MIT licensed, and the source is on GitHub. Tests run with bun test.