latitude llm - Moses Yebei

The Problem

Teams building LLM features in production lack a single place to see what their prompts actually return, judge whether those outputs are good, and fix the ones that aren't. Instrumentation, prompt versioning, evaluation, and deployment typically live in separate tools, so a regression in one prompt is found days later in a ticket, not in CI.

What This Does

Latitude is an open-source AI engineering platform: capture prompts, inputs, outputs, tool calls, latency, and token cost from real traffic; replay those runs in a prompt playground; curate datasets and evals; and publish versioned prompts to a gateway. The repo is a collection of six self-contained projects, not one codebase — packages/ (2,581 files) holds the core libraries, apps/ (1,688 files) holds the web console, gateway, engine, and CLI, plus examples/, tools/, docker/, and bin/. It is a fork of latitude-dev/latitude-llm (4,603 stars) with no stars of its own.

How It Is Wired

The call graph maps 42 resolved internal edges. The widest blast radius sits in packages/web-ui/src/lib/utils.ts — cn is called from 10 places, so changing its signature breaks the most. hashContent (packages/cli/src/utils/hashContent.ts) and buildProviderLogResponse (packages/core/src/services/providerLogs/buildResponse.ts) are each called from 3 places. The graph is shallow: entry points like configureWebhookRoutes (apps/gateway/src/routes/webhook/index.ts:11) reach exactly one function, so most changes are locally contained.

The wiring beyond internal calls is thin. One function touches a database (analyzeQuery in packages/core/src/lib/analyzeQuery.ts), and two perform cryptographic work (hashPassword, verifyPassword in packages/core/src/lib/password.ts). The gateway entry (apps/gateway/src/openApi/createApp.ts) builds routers via createRouter, and the Python engine (apps/engine/app/main.py) runs an RPC server (apps/engine/app/rpc/server.py). The repository is a portfolio: the web app, gateway, Python engine, and CLI each have their own entry point and lifecycle.

How To Use It

The README documents setup but the repo gives no root-level install command. Evidence points to pnpm (lockfile present) for the TypeScript projects and apps/engine/pyproject.toml for the Python engine. Configuration belongs in .env.example at the root. Dockerfiles exist (apps/gateway/docker/Dockerfile), so a container build is viable. The README's quick start covers cloud signup and self-hosted deployment; follow those steps verbatim rather than guessing flags.

git clone https://github.com/moses-y/latitude-llm
# then follow README.md quick start for self-hosted setup

Real-World Use

Instrument existing LLM calls with the telemetry SDK, watch failures cluster in the console, convert an issue into an automatic eval, and publish a fixed prompt to the gateway — the reliability loop the README describes. The examples/ folder (74 files) holds reference integrations.

Code Health & Issues

Static analysis found 9 findings (0 critical, 2 high, 6 medium, 1 low):

High – Third-party GitHub Actions pinned to mutable tags (pnpm/action-setup@v2, softprops/action-gh-release@v2) in .github/workflows; pin to commit SHAs.
High – CI never runs the test suite: 563 test files exist, but no test command appears in the 4 workflows read.
Medium – GITHUB_TOKEN permissions undeclared in rollback-deployment.yml; add permissions: contents: read.
Medium – CI uses npm install with a committed lockfile; use frozen-lockfile installs.
Medium – Base image node:22-alpine unpinned in apps/gateway/docker/Dockerfile; pin by digest.
Medium – No dependency vulnerability scan in CI.
Medium – Two blobs over 5MB (gif_ui.gif at 12.3MB and 5.7MB); move to LFS.
Medium – checkout keeps credentials in publish-cli.yml; set persist-credentials: false.
Low – No timeout-minutes on workflow jobs.

The audit also flags committed secrets present in the repo — verify charts/latitude/values.secrets.yaml.example against your threat model before deploying.

The Bottom Line

A genuinely useful prompt-engineering platform with real substance — observability, evals, a gateway, and a Python optimization engine — but the fork carries untested CI and committed-secret risk. Use it for the product; fix the two high findings before you trust a green check.

What the analyser found

Deployment readiness

6/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

4,729 files

TypeScript2500
TSX939
JSON317
SQL268
Python134
Markdown48

ReactNext.jsExpressTailwindDockerKubernetesTransformersLangChainVector storeNeo4j

Findings

LowSecret-shaped paths present; the code-health audit confirms or clears them charts/latitude/values.secrets.yaml.example, packages/core/src/assets/eu-central-1-bundle.pem

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high2

medium6

low1

Pin third-party GitHub Actions to a commit SHAhigh3 occurrences

.github/workflows

pnpm/action-setup@v2, softprops/action-gh-release@v2, aws-actions/configure-aws-credentials@v4.0.2

A tag can be moved, so the action running with your token and secrets is whatever its owner last pushed; this is how tj-actions/changed-files leaked secrets from thousands of repos.

Fix: Replace each @vN with the 40-character commit SHA, keep # vN as a comment, and let Dependabot bump the SHAs.

Make CI invoke the test suite it hashigh563 occurrences

.github/workflows

563 test files, no test command in the 4 workflows read

A green check that never executed an assertion is worse than no check, because reviewers trust it.

Fix: Add a test step to the existing workflow rather than a new workflow.

Declare least-privilege permissions for GITHUB_TOKENmedium

.github/workflows/rollback-deployment.yml

1 workflow(s) declare no permissions, 1 of them reference secrets

With no declaration the token inherits the repository default, so any injected step can push commits or mint releases from inside your own CI.

Fix: Add permissions: contents: read at the top of the workflow and widen per job only where needed.

Install from the lockfile in CImedium

.github/workflows

npm install with a committed package-lock.json

A fresh resolution in CI means the tested dependency set is not the locked one, so the failure only appears after merge.

Fix: Use npm ci, yarn install --immutable, or pnpm install --frozen-lockfile.

Pin the container base image by digestmedium2 occurrences

apps/gateway/docker/Dockerfile

node:22-alpine, node:22-alpine

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Gate pull requests on a dependency vulnerability scanmedium

.github/workflows

no dependency scan in CI

This is the one gate that would catch a known-vulnerable package before it reaches a build, and no repository in the sample had it.

Fix: Add dependency-review-action on pull_request, or osv-scanner on push and a schedule.

Move large binaries to Git LFS or out of the repositorymedium2 occurrences

docs/assets/readme/gif_ui.gif

2 blobs over 5MB: gif_ui.gif 12.3MB, gif_ui.gif 5.7MB

One repository here carries twenty blobs over 5MB including a 10.9MB spreadsheet, so every clone and every CI checkout pays for data nobody diffs.

Fix: Track those extensions with LFS, or move datasets to object storage and fetch them in a setup step.

Set persist-credentials: false on checkoutmedium

.github/workflows/publish-cli.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Set timeout-minutes on the workflow jobslow4 occurrences

.github/workflows/cicd.yml

4 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.