Turned 300 university websites into one clean program database for a student recruitment agency
Client snapshot
A student recruitment agency with 10 counsellors placing international students into master's programs. They work with about 300 universities across four countries.
The problem
Counsellors needed one list of programs with the same fields for each: tuition, intake dates, entry requirements, English test minimums, application deadlines and a contact. Every university's site laid this out differently. An intern rebuilt the list by hand once a year, it took about three weeks, and it was out of date by the next intake. Counsellors kept their own notes on top, so nobody trusted the master list.
What we built
A crawler and an extractor that write to one sheet.
The agency keeps a sheet of university URLs. n8n sends each one to a Python crawler built on Playwright, which follows the program pages and pulls the text. Claude reads each program page and fills a fixed schema: program name, degree, tuition, intakes, deadlines, entry and English requirements, contact. When a field isn't on the page, it's left blank and flagged, not guessed.
Duplicates across pages get merged. Each program is scored against the agency's placement criteria, so counsellors see the strongest matches first. Everything lands in one Google Sheet with a source link per row, so any figure can be checked in a click.
The run repeats at the start of every term. Rows where a deadline or tuition figure changed are highlighted.
Results, first term
- Building the full program list: about three weeks of intern time → two days, mostly reviewing flagged rows.
- Fields filled per program: around 60% → over 95%, each with a source link.
- Refreshed every term instead of once a year, with changes highlighted.
- Counsellors dropped their private lists and work from the shared sheet.
Quote
"The deadlines are right now, and when one changes I can see it changed. That alone paid for it." Agency director
Build snapshot
- Timeline: 5 business days
- Stack: n8n, Python, Playwright, Claude, Google Sheets
- Investment: $1,800
- Who owned it: the operations coordinator runs the termly refresh and reviews the flagged rows.
The pattern
This is the pattern we ship in the Lead Scraper + Enricher engagement: a list of URLs in, a structured, scored list out. Here the rows were programs instead of companies. The build is the same, with your schema swapped in.
This is an illustrative composite drawn from operator engagements. Names, identifying details, and exact metrics are anonymised. The workflow, stack, and approach are real.
Looks like your situation?
A 30-minute call tells us whether the same pattern fits your operation, and what it would cost to ship it.