The Problem
LLMs that need up‑to‑date web data must contend with dynamic page structures, authentication flows, and rate‑limited APIs. Building a reliable scraper for each site quickly becomes a maintenance burden, especially when UI changes break hard‑coded selectors.
What This Does
AgentQL provides a natural‑language‑driven query language that translates into Playwright selectors, allowing scripts to locate elements by their visible text or context instead of brittle CSS/XPath strings. The repository ships the Python SDK (examples/python/.../main.py) and JavaScript SDK (examples/js/.../main.js) that wrap Playwright, a REST API (defined in the documentation, not in this fork), and a browser‑extension debugger for interactive query tuning.
The folder layout makes the intent clear:
examples/python/ – ready‑to‑run .py scripts that import agentql and execute queries. examples/js/ – equivalent Node scripts with a package.json that pulls in the JS SDK. .templates/ – starter files for new projects (templateasync.py, templatesync.py).
Key files:
examples/python/pyproject.toml – Poetry‑managed dependency list, including playwright and agentql. examples/js/package.json – NPM manifest that lists @agentql/sdk and playwright. README.md – high‑level overview and quick‑start links.
How To Use It
Install the SDK
Python (run in the examples/python folder):
cd examples/python poetry install # reads pyproject.toml, installs playwright & agentql poetry run python main.py # runs the example script
JavaScript (run in the examples/js folder):
cd examples/js npm ci # installs from package-lock.json node close-cookie-dialog/main.js # runs the chosen example Configure credentials (if needed) Scripts that access authenticated sites expect a .env file or environment variables named AGENTQLAPIKEY and AGENTQLENDPOINT. The repository does not ship a sample, so create one according to the SDK docs. Run a query Each main. file builds an AgentQL query string (e.g., "list all product titles where price < $20"), calls agentql.run(query), and prints structured JSON. Modify the query or the CSV templates in list-query-usage/productsdata.csv to suit your data model. Debug with the browser extension Install the Chrome extension from the README link, open a target page, and use the extension UI to test selectors before embedding them in a script.
Real‑World Use
A marketing analytics pipeline can pull daily pricing tables from competitor sites:
from agentql import AgentQL
client = AgentQL(apikey=os.getenv("AGENTQLAPIKEY")) query = """ list product name, price, availability from https://competitor.com/catalog where price < $50 """ data = client.run(query) feed data into a pandas DataFrame for downstream reporting
The same query works on a different retailer without code changes, thanks to the natural‑language selector engine.
Code Health & Issues
Low – Missing test suite – No tests/ directory; CI workflows run only linters and secret scanners. Medium – Core library absent – This fork contains only examples and templates; the actual agentql package resides in the upstream repo. Users must install it from PyPI (pip install agentql). Low – License present – LICENSE file included, so reuse is permitted. Medium – Environment‑variable assumptions – Scripts reference AGENTQLAPI_KEY but no .env.example is provided, risking runtime errors. Low – Dependency hygiene – Both poetry.lock and package-lock.json are committed, ensuring reproducible builds. Low – CI coverage – GitHub Actions run pre‑commit checks and vulnerability scans, but no automated test execution.
The Bottom Line
AgentQL offers a practical abstraction over Playwright, letting developers write high‑level, language‑driven extraction scripts. The repository is well‑organized for rapid prototyping, but it lacks internal tests and the core SDK source, so production teams should treat it as a reference implementation and rely on the published package for stability. Suitable for teams that need to prototype web‑data pipelines quickly and are comfortable adding their own test coverage.