Architecture & Key Constraints
The dashboard and the Puppeteer API are separate apps. Zod checks the request, and Puppeteer tests drive the workflow builder.
The product
ScrapeAutomate is a monorepo with a marketing site, a Next.js dashboard, and an Elysia API. The API renders pages with Puppeteer and puppeteer-extra. Zod checks the request body. Docker Compose runs PostgreSQL, Redis, and MinIO next to the apps. The dashboard uses Prisma for PostgreSQL and ioredis for Redis.
From the dashboard you keep API keys and saved workflows. A workflow can be created in the builder, run, searched, and archived.
Tests
The repository drives that builder with Puppeteer. The tests create a workflow, check that an empty form shows validation errors, run a workflow, archive one, and search the list.
My part
I worked as a frontend and API engineer on the dashboard and the scraper API, including rate limiting with rate-limiter-flexible, route validation, and the workflow builder. That is the work a web scraping API needs to be usable: requests stay within limits, bad input is rejected before it reaches the browser, and saved workflows can be run again from the dashboard.



