GlobeScraper
A full-stack content, community, and rental data platform for English teachers relocating to Southeast Asia. Built from scratch with Next.js 14, a 970-line Prisma schema, 7-source scraping pipeline, and AI content engine powered by Google Gemini.
API routes
55
DB models
30+
Scraper sources
7
Deploy
Vercel + Hetzner
How it works
Rental Pipeline
Discover phase crawls category pages and enqueues URLs. Workers atomically claim items (SQL UPDATE…LIMIT, no locks), fetch and parse content, then upsert with content fingerprinting to prevent duplicates. AI review stage uses Gemini to classify residential vs non-residential, correct property types, and rewrite descriptions.
Platform Features
Rental marketplace with search filters, pagination, and image carousels. Community features: profiles, connections, DMs, meetups, trust panels, and moderation. AI content engine researches competitors via Serper.dev, generates articles with Gemini, creates images with Imagen 4.0, and auto-publishes with SEO scoring. Analytics heatmap covering 300+ Cambodia districts.
Key Decisions
No Tailwind - vanilla CSS + BEM for full control. Browser automation is reserved for permitted public sources where an API is unavailable, with source terms reviewed before collection. Conservative pacing and bounded concurrency reduce load. Atomic queue claiming prevents race conditions. Gemini was selected for speed, cost, and structured output; a VPS supports long-running browser tasks that do not suit serverless execution.
Security
- Source permissions, terms, and robots guidance are reviewed before collection
- Conservative rate limits and bounded concurrency reduce source load
- Content fingerprinting prevents duplicate records
- Authentication via Auth.js v5
- Atomic queue claiming prevents race conditions
Related service
This project demonstrates the kind of work I do under Custom AI Solutions.
Next step / direct enquiry
Want to discuss the engineering behind this work?
Send a private message about a role, collaboration, or technical problem.