The Problem

Many professionals struggle to evaluate and synthesize responses from multiple Large Language Model providers for complex queries. Typically, users rely on a single model or manually copy-prompt several models, losing the benefits of comparative analysis, built-in review cycles, and ranked consensus building that a structured council provides.

What This Does

This repository implements a "LLM Council" workflow as a local web application. The core logic resides in backend/council.py, which orchestrates a three-stage process: Stage 1 (stage1_collect_responses) dispatches a user query to all configured models via backend/openrouter.py using the query_models_parallel function, collecting individual responses. Stage 2 (stage2_collect_rankings) anonymizes these responses and asks each model to rank the others for accuracy and insight, utilizing parse_ranking_from_text to process the output. Stage 3 (stage3_synthesize_final) has a designated Chairman model compile the ranked inputs into a single final answer. The frontend, built with React and Vite in frontend/src/, presents these stages in a tabbed interface, allowing users to inspect individual model outputs and the final synthesized result.

How It Is Wired

Execution begins at the send_message entry point in backend/main.py:83, which reaches 16 functions and is called by nothing else in the repository. The call graph shows send_message triggers run_full_council, which sequentially calls stage1_collect_responses, stage2_collect_rankings, calculate_aggregate_rankings, and stage3_synthesize_final. Data flows through backend/storage.py, which defines ensure_data_dir and manages JSON file persistence in data/conversations/. The system makes one outbound network call via the path send_message -> generate_conversation_title -> query_model [network via client.post] to OpenRouter. backend/main.py also defines the FastAPI routes and data models (CreateConversationRequest, SendMessageRequest), while backend/openrouter.py handles the specific HTTP communication with the API.

How To Use It

Setup:

  • Install backend dependencies: uv sync
  • Install frontend dependencies: cd frontend && npm install && cd ..

Configuration:

  • Create a .env file in the project root containing OPENROUTER_API_KEY=sk-or-v1-....
  • Optionally edit backend/config.py to modify COUNCIL_MODELS or CHAIRMAN_MODEL.

Running it:

  • Option 1: Run ./start.sh.
  • Option 2: Start the backend in one terminal: uv run python -m backend.main. Start the frontend in another: cd frontend && npm run dev.
  • Open http://localhost:5173 in a browser.

Real-World Use

A user submits a complex research question, such as "Compare the architectural differences between React Server Components and traditional Node.js endpoints for real-time data." The LLM Council dispatches this to four models simultaneously. The user sees four distinct opinions side-by-side. The models then review each other's work anonymously, ranking accuracy. Finally, the Chairman model synthesizes the top-ranked insights into a cohesive answer, which the user can accept or iterate upon.

Code Health & Issues

  • [HIGH] Add a LICENSE; redistribution rights are undefined — evidence: no licence file at the repository root. With no licence the default is all rights reserved, so this code cannot legally be reused.
  • [HIGH] Add a test suite; this repository has none — evidence: 17 source files, no test files. Any change ships with no signal that existing behaviour still holds.
  • [HIGH] Add a workflow that builds and tests this repository — evidence: 17 source files, no CI configuration. Every change merges with nobody having run the build once.
  • [MEDIUM] Enable Dependabot or Renovate — evidence: 2 manifest(s), no update bot configured. Without a bot a published advisory sits unpatched until someone audits by hand.

The Bottom Line

This is a functional, educational prototype that successfully demonstrates a multi-LLM ranking and synthesis workflow. The code is clean and the three-stage council pattern is clearly implemented. However, the absence of a license, tests, and CI makes it unsuitable for production use without significant guardrails. It is best suited for experimentation, evaluation, or as a reference architecture for building localized LLM orchestration tools.