The Problem
Many organizations need to extract structured data from public websites, often behind bot‑detectors, captchas, or even .onion services. Building a custom scraper that can bypass these protections, handle dynamic content, and produce clean exports is time‑consuming and error‑prone.
What This Does
CyberScraper‑2077 combines a Streamlit front‑end (app/streamlitwebscraperchat.py) with a set of Python scrapers (src/scrapers/). The core logic lives in src/httpclient.py, src/models.py, and the LLM prompt handling in src/prompts.py.
AI‑driven parsing is implemented in src/ollamamodels.py and src/models.py, which call OpenAI, Gemini, or local Ollama models to transform raw HTML into JSON/CSV/etc. Dynamic page rendering is handled by src/scrapers/playwrightscraper.py, while Tor support lives under src/scrapers/tor/ (e.g., tormanager.py, torscraper.py). Export utilities (src/utils/googlesheetsutils.py) and caching (src/utils/errorhandler.py) are wired into the Streamlit UI.
How To Use It
Setup
git clone https://github.com/itsOwen/CyberScraper-2077.git cd CyberScraper-2077 python -m venv venv source venv/bin/activate # Windows: venv\Scripts\activate pip install -r requirements.txt playwright install
Configuration
Set the required API keys in the environment before launching:
export OPENAIAPIKEY="your-openai-key" export GOOGLEAPIKEY="your-gemini-key" Optional for Google Sheets uploads export GOOGLESHEETSCREDENTIALS="/path/to/credentials.json"
The Streamlit UI reads these variables at runtime (see app/utils.py for loading logic).
Running
Local development: streamlit run app/streamlitwebscraperchat.py
The UI invokes src/webextractor.py which selects the appropriate scraper class (Playwright, Tor, etc.) based on user input.
Docker*: docker build -t cyberscraper . docker run -e OPENAIAPIKEY=... -e GOOGLEAPIKEY=... -p 8501:8501 cyberscraper
The Dockerfile copies the source tree, installs requirements.txt, and runs the Streamlit entry point.
Real‑World Use
A market‑research team could schedule a daily run (via cron or a CI job) that launches the Docker container, points the scraper at a competitor’s product catalog, and pushes the resulting CSV to a shared Google Sheet. Example snippet for automation:
docker run --rm \ -e OPENAIAPIKEY=$OPENAIAPIKEY \ -e GOOGLEAPIKEY=$GOOGLEAPIKEY \ cyberscraper \ python src/webextractor.py --url "https://example.com/products" --output sheet
Code Health & Issues
Medium – No automated tests – repository lacks any tests/ directory or pytest configuration. Untested code paths increase regression risk. Medium – No CI/CD pipeline – .github/ contains only issue templates and funding info; no GitHub Actions or other CI config. Low – No lockfile – dependencies are listed only in requirements.txt; reproducible builds depend on PyPI state. Low – Potential secret leakage – README mentions environment variables but no .env.example or guidance on secure storage. Low – Limited documentation – three Markdown docs exist, but no explicit API reference for scraper classes; developers must read source files. Low – Docker build assumes host‑level network access – Tor usage may require additional capabilities not documented in Dockerfile.
No obvious security‑critical bugs are visible, but the lack of input validation in src/scrapers/base_scraper.py could allow malformed URLs to trigger unexpected errors.
The Bottom Line
CyberScraper‑2077 offers a ready‑made, AI‑augmented scraping stack with Tor and Streamlit integration, suitable for teams that need rapid prototyping of complex extraction tasks. However, the absence of tests, CI, and a lockfile means the codebase requires additional engineering effort to reach production‑grade reliability. It is a good fit for small‑to‑medium projects where the AI‑driven parsing value outweighs the operational overhead of adding proper testing and CI pipelines.