CyberScraper 2077 - Moses Yebei

The Problem

Extracting structured data from arbitrary web pages usually means writing per-site selectors that break when the target changes. CyberScraper-2077 replaces that with an LLM that reads the page and returns the fields you asked for, in the format you specify.

What This Does

A Streamlit app (app/streamlit_web_scraper_chat.py) that takes a URL and a prompt, scrapes the page via Playwright or Tor, and sends the content to OpenAI, Gemini, or a local Ollama model for extraction. Results export as JSON, CSV, HTML, SQL, or Excel, and can upload to Google Sheets via src/utils/google_sheets_utils.py.

The scraper layer lives in src/scrapers/: base_scraper.py defines the abstract contract, playwright_scraper.py handles browser automation with stealth settings, and tor/ provides onion routing via tor_manager.py and tor_scraper.py. Extraction logic sits in src/web_extractor.py, which handles URL normalization, content fetching, and token counting.

How It Is Wired

Execution starts at main in main.py:415, which reaches 81 functions. The UI layer calls process_message, which routes through process_with_progress → process_query → _chat_without_content, ending in a model inference call (self.model.generate). The Ollama path is main → list_ollama_models → list_models, which makes a network call via session.get.

The hub is src/web_extractor.py (Ca 2, Ce 8, instability 0.8) — 25 functions that everything routes through. src/utils/error_handler.py is the stable core (Ca 7, Ce 0, instability 0): seven modules import it, it imports nothing. _extract_json_from_markdown is called from 6 places, so changing its signature breaks the most code. src/scrapers/playwright_scraper.py carries the heaviest load: 27 functions, 694 lines, and it launches an external browser process.

No import cycles detected, which keeps refactoring tractable. The main risk is playwright_scraper.py — oversized, deeply nested (indentation depth 8), and called from 3 other files.

How To Use It

git clone https://github.com/moses-y/CyberScraper-2077.git
cd CyberScraper-2077
virtualenv venv && source venv/bin/activate
pip install -r requirements.txt
playwright install
export OPENAI_API_KEY="your-api-key-here"
export GOOGLE_API_KEY="your-api-key-here"
python main.py

Requires Python 3.10+. For Ollama, install Ollama locally and select a model in the UI. A Dockerfile exists for containerized runs, but the README notes the captcha-bypass feature doesn't work under Docker.

Real-World Use

A data analyst scraping a product page: paste the URL, ask for "price, name, and availability," and get a CSV back. The Tor path (src/scrapers/tor/) handles .onion sites with automatic circuit setup, so the same workflow extends to sources that block datacenter IPs. The caching layer (LRU + custom dict) reduces redundant LLM calls for repeated queries.

Code Health & Issues

Static analysis (measured, not opinion) found 11 issues: 2 high, 9 medium. High: deep nesting (depth 8) in src/ollama_models.py, src/scrapers/playwright_scraper.py, and src/web_extractor.py. Medium: oversized playwright_scraper.py (694 lines), broad exception handling in src/scrapers/tor/utils.py and main.py, duplicated blocks across base_scraper.py and playwright_scraper.py, and high branching density in src/web_extractor.py (112 branch points over 393 lines).

SDLC gaps: no test files, no CI, no lockfile (requirements.txt unpinned), Dockerfile uses a mutable python:3.12-slim-bookworm tag and runs as root, and no Dependabot. Any change ships with zero regression signal.

The Bottom Line

Functionally interesting — LLM-driven extraction beats brittle selectors, and the Tor support is a differentiator. But it's a fork with no tests, no CI, and a 694-line core module that needs splitting before it's safe to extend. Use it for prototyping scraping workflows, not for production data pipelines.

What the analyser found

Deployment readiness

4/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

38 files

Python22
Markdown6
YAML1
CSS1

Docker

Architecture

Top-level areas of the codebase, sized by module count. Arrows show how many imports cross from one area into another.

graph TD g0["src<br/>17 modules"] g1["app<br/>3 modules"] g2["(root)<br/>2 modules"] g2 -->|5| g0 g2 -->|3| g1 g1 -->|2| g0

Module dependencies

The 10 most depended-upon modules of 22, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_src_utils_error_handler["error_handler"] n1__scrapers_playwright_scraper["playwright_scraper"] n2_src_scrapers_tor_tor_config["tor_config"] n3_src_web_extractor["web_extractor"] n4_src_ollama_models["ollama_models"] n5_src_scrapers_base_scraper["base_scraper"] n6_src_scrapers_tor_exceptions["exceptions"] n7_src_scrapers_tor_utils["utils"] n8_src_scrapers_tor_tor_scraper["tor_scraper"] n9_src_scrapers_tor_tor_manager["tor_manager"] n4_src_ollama_models --> n0_src_utils_error_handler n1__scrapers_playwright_scraper --> n5_src_scrapers_base_scraper n1__scrapers_playwright_scraper --> n0_src_utils_error_handler n9_src_scrapers_tor_tor_manager --> n2_src_scrapers_tor_tor_config n9_src_scrapers_tor_tor_manager --> n7_src_scrapers_tor_utils n9_src_scrapers_tor_tor_manager --> n6_src_scrapers_tor_exceptions n9_src_scrapers_tor_tor_manager --> n0_src_utils_error_handler n8_src_scrapers_tor_tor_scraper --> n9_src_scrapers_tor_tor_manager n8_src_scrapers_tor_tor_scraper --> n2_src_scrapers_tor_tor_config n8_src_scrapers_tor_tor_scraper --> n7_src_scrapers_tor_utils n8_src_scrapers_tor_tor_scraper --> n5_src_scrapers_base_scraper n8_src_scrapers_tor_tor_scraper --> n0_src_utils_error_handler n3_src_web_extractor --> n4_src_ollama_models n3_src_web_extractor --> n1__scrapers_playwright_scraper n3_src_web_extractor --> n0_src_utils_error_handler n3_src_web_extractor --> n8_src_scrapers_tor_tor_scraper n3_src_web_extractor --> n2_src_scrapers_tor_tor_config n3_src_web_extractor --> n6_src_scrapers_tor_exceptions

Findings

MediumNo test files detected - untested code paths repository-wide
MediumNo CI/CD pipeline detected - no automated build/test gate .github/ or CI config
LowDependencies declared without a lockfile - non-reproducible builds requirements.txt

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high3

medium3

Add a test suite; this repository has nonehigh22 occurrences

22 source files, no test files

Any change ships with no signal that existing behaviour still holds, so a regression reaches production undetected.

Fix: Add one test per public entry point, then a CI step that runs them.

Add a workflow that builds and tests this repositoryhigh

22 source files, no CI configuration

Every change merges with nobody having run the build once, which across this estate is 499 repositories.

Fix: Add a workflow running the project build and test command on push and pull_request.

Add a build gate for the deployable artifacts herehigh

Dockerfile

deployment artifacts with no workflow

Something in the repo is meant to reach a server and no automated step has ever validated it, which is the state of 125 repositories here.

Fix: Add a workflow that at least builds the image and validates the manifests.

Enable Dependabot or Renovatemedium

1 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Pin the container base image by digestmedium

Dockerfile

python:3.12-slim-bookworm

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Add a non-root USER to the imagemedium

Dockerfile

CMD or ENTRYPOINT with no USER directive

A process running as root in the container is root against every mounted volume, and it turns any container escape or writable-mount mistake from a contained problem into a host one.

Fix: Create an unprivileged user, chown what it needs, and end the Dockerfile with USER.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.