agentql

AgentQL – Technical Briefing Repository: https://github.com/moses-y/agentql (clone with git clone https://github.com/moses-y/agentql)

The Problem

Automation scripts that scrape or interact with modern web sites must handle dynamic content, authentication, and UI‑drift. Maintaining separate Playwright code for each site quickly becomes brittle and duplicated.

What This Does

AgentQL supplies a query language that lets an LLM describe the data to extract in natural language. The language is interpreted by SDKs that drive Playwright, so the same query works across pages that share layout or content. Key artefacts

The SDKs translate a query (e.g. “list product name and price”) into Playwright actions: locate elements, click, scroll, and return structured JSON.

How It Is Wired

1. Entry point – a concrete script such as examples/js/close-popup/main.js (or the Python equivalent) runs main at line 15.
2. main calls a small helper chain (fetchData, extractPricingData, etc.). The internal call graph shows 25 resolved edges; the most‑used functions are fetchData and goToTheNextPage, each invoked from two places.
3. The helper functions perform the only side‑effects: fs.writeFileSync in fetchData (writes scraped JSON to disk). A model call (e.g., OpenAI) in perform-sentiment-analysis/main.js (imports a model for inference).
4. No circular imports are present; each module is isolated, which simplifies refactoring.
5. The templates (.templates/js/template.js) provide the skeleton that each example copies, explaining why the same 6‑line blocks appear in 23 files – a high‑severity DRY issue.

Thus a typical run follows:

main → fetchData → (fs.writeFileSync)  
main → extractPricingData → goToTheNextPage → fetchData

Only the listed functions touch the filesystem or external AI services; all other code stays within Playwright and the query language.

How To Use It

# 1. Clone the repo
git clone https://github.com/moses-y/agentql
cd agentql

# 2. Install JavaScript example dependencies (uses lockfile)
npm ci   # runs in examples/js/

# 3. Install Python example dependencies (Poetry lockfile)
poetry install   # runs in examples/python/

Run a demo – pick any example folder and execute its main script:

# JavaScript example
node examples/js/collect-pricing-data/main.js

# Python example
python examples/python/collect_ecommerce_pricing_data/main.py

No additional configuration files (e.g., .env) are committed; any required API keys must be supplied at runtime as documented in the SDK readmes.

Real‑World Use

A SaaS platform that needs nightly price monitoring could store a query like:

{
  "select": ["product_name", "price"],
  "where": "category = 'laptops'"
}

The platform calls the Python SDK (agentql.run_query(query)) inside a scheduled Celery task. The SDK drives Playwright, writes the result to a JSON file (fs.writeFileSync path), and the platform ingests that file for downstream analytics.

Code Health & Issues

• High – GitHub Actions reference mutable tags (snok/install-poetry@v1, isort/isort-action@master, etc.). Pin each to a commit SHA.
• Medium – Workflows lack explicit permissions for GITHUB_TOKEN; add permissions: { contents: read }.
• Medium – CI uses npm install instead of npm ci; replace to guarantee lockfile fidelity.
• Medium – Checkout step keeps credentials; set persist-credentials: false.
• Low – No timeout-minutes on workflow jobs; define a reasonable bound (e.g., timeout-minutes: 30).

Additional observations from the static audit:

• No test files are present; all example scripts run untested.
• The codebase contains duplicated 6‑line blocks across 23 example files – extract shared helpers into a utility module.
• Broad except clauses in examples/python/collect_ecommerce_pricing_data/main.py swallow errors; narrow exception handling is recommended.
• High branching density in examples/python/news-aggregator/main_sync.py suggests refactoring into smaller strategy functions.

The Bottom Line

AgentQL delivers ready‑to‑run Playwright examples that showcase a natural‑language query layer, making web automation accessible to LLM‑driven agents. The repository is well‑structured but consists largely of duplicated, untested examples, and the CI configuration needs hardening. Teams comfortable with Playwright can adopt the SDKs quickly, but should add proper tests and refactor shared logic before production use.