(llm-first): context budget, validation, and eval harness, orchestration general-prompt
This commit is contained in:
MaKarin
237
docs/generation-playbook.md
Normal file
237
docs/generation-playbook.md
Normal file
@@ -0,0 +1,237 @@
|
||||
# Generation Playbook
|
||||
|
||||
Step-by-step instructions for generating and regenerating artifacts in this repository.
|
||||
Read `AGENTS.md` and `docs/source-of-truth.md` first.
|
||||
|
||||
---
|
||||
|
||||
## Pipeline overview
|
||||
|
||||
```
|
||||
Tier 1 — Single Source of Truth (hand-authored, never generated)
|
||||
domain/toir.api.dsl ──┐ enums, DTOs, endpoints, HTTP methods,
|
||||
│ entity field mappings, primary keys
|
||||
│
|
||||
▼
|
||||
Tier 2 — Deterministic Preprocessing (npm scripts, no LLM)
|
||||
api-summary.json ←─ npm run generate:api-summary
|
||||
openapi.json ←─ npm run generate:openapi (auxiliary)
|
||||
│
|
||||
▼
|
||||
Tier 1 (api.dsl) + Tier 2 (context) + prompts/*.md ──► LLM Generation
|
||||
prompts/general-prompt.md
|
||||
prompts/backend-rules.md ──► server/src/modules/<entity>/
|
||||
prompts/frontend-rules.md ──► client/src/resources/<entity>/
|
||||
prompts/prisma-rules.md ──► server/prisma/schema.prisma
|
||||
prompts/auth-rules.md ──► (auth seam reference)
|
||||
prompts/runtime-rules.md ──► (env/docker reference)
|
||||
prompts/validation-rules.md ──► (validation gate reference)
|
||||
│
|
||||
▼
|
||||
Tier 4 — Validation Gate
|
||||
node tools/validate-generation.mjs --artifacts-only
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before any generation run:
|
||||
|
||||
1. `domain/*.api.dsl` is current and valid.
|
||||
2. Refresh the Tier 2 intermediate context:
|
||||
```bash
|
||||
npm run generate:api-summary
|
||||
```
|
||||
3. Run `node tools/validate-generation.mjs --artifacts-only` to confirm the baseline passes.
|
||||
|
||||
---
|
||||
|
||||
## Standard generation workflow
|
||||
|
||||
### Step 1 — Refresh Tier 2 derived artifacts
|
||||
|
||||
```bash
|
||||
# From repo root
|
||||
npm run generate:api-summary
|
||||
```
|
||||
|
||||
### Step 2 — Read generation inputs (context budget)
|
||||
|
||||
> **`prompts/general-prompt.md` is the master generation prompt.** It contains all core
|
||||
> type mappings, naming conventions, DTO/controller/service/frontend rules, mutation
|
||||
> boundaries, and the complete generation workflow. Load it as the single entrypoint.
|
||||
>
|
||||
> For artifact-specific details (Prisma FK rules, auth JWKS chain, detailed validation
|
||||
> groups), load the relevant companion file: `prompts/prisma-rules.md`,
|
||||
> `prompts/auth-rules.md`, `prompts/validation-rules.md`, or `prompts/runtime-rules.md`.
|
||||
>
|
||||
> See `prompts/general-prompt.md §CONTEXT BUDGET` for the full budget model.
|
||||
|
||||
1. `prompts/general-prompt.md` — master generation prompt (always load)
|
||||
2. `api-summary.json §<entity>` — compact entity index (fast-path context anchor)
|
||||
3. `domain/*.api.dsl §API.<EntityName>` — **only the api block + its referenced DTOs + enums** (entity-scoped)
|
||||
4. **If needed:** `prompts/prisma-rules.md` (Prisma) or `prompts/auth-rules.md` (auth seams)
|
||||
|
||||
Before generating any DTO or component: **quote the relevant DSL field definitions verbatim first**, then generate from those quotes. This prevents training-data contamination.
|
||||
|
||||
### Step 3 — Generate Prisma schema
|
||||
|
||||
Generate `server/prisma/schema.prisma` from `domain/*.api.dsl` following `prompts/prisma-rules.md`.
|
||||
|
||||
If the schema changed, run Prisma migration in `server/`:
|
||||
|
||||
```bash
|
||||
cd server
|
||||
npx prisma migrate dev --name <description>
|
||||
```
|
||||
|
||||
### Step 4 — Generate backend modules
|
||||
|
||||
For each `api` block in `domain/*.api.dsl`, generate:
|
||||
|
||||
1. `server/src/modules/<kebab>/<entity>.module.ts`
|
||||
2. `server/src/modules/<kebab>/dto/create-<kebab>.dto.ts`
|
||||
— fields from the `DTO.<Entity>Create` block in api.dsl
|
||||
3. `server/src/modules/<kebab>/dto/update-<kebab>.dto.ts`
|
||||
— fields from the `DTO.<Entity>Update` block in api.dsl
|
||||
4. `server/src/modules/<kebab>/<entity>.service.ts`
|
||||
— CRUD operations using Prisma; respect type mappings from `prompts/backend-rules.md`
|
||||
5. `server/src/modules/<kebab>/<entity>.controller.ts`
|
||||
— one method per `endpoint` in the `api` block; HTTP verb and path from the `label`
|
||||
|
||||
Register the module in `server/src/app.module.ts`.
|
||||
|
||||
### Step 5 — Generate frontend resources
|
||||
|
||||
For each `api` block in `domain/*.api.dsl`, generate:
|
||||
|
||||
1. `client/src/resources/<kebab>/<Entity>List.tsx`
|
||||
— columns from `DTO.<Entity>` (response shape)
|
||||
2. `client/src/resources/<kebab>/<Entity>Create.tsx`
|
||||
— fields from `DTO.<Entity>Create`
|
||||
3. `client/src/resources/<kebab>/<Entity>Edit.tsx`
|
||||
— fields from `DTO.<Entity>Update`
|
||||
4. `client/src/resources/<kebab>/<Entity>Show.tsx`
|
||||
— fields from `DTO.<Entity>`
|
||||
|
||||
Register the resource in `client/src/App.tsx`.
|
||||
|
||||
### Step 6 — Verify (two-stage gate)
|
||||
|
||||
**Stage 1 — Structural gate:**
|
||||
|
||||
```bash
|
||||
node tools/validate-generation.mjs --artifacts-only
|
||||
```
|
||||
|
||||
Checks file existence, field names, class-validator decorators, auth guards, and RA component types.
|
||||
|
||||
Full structural verification (requires installed deps):
|
||||
|
||||
```bash
|
||||
node tools/validate-generation.mjs
|
||||
```
|
||||
|
||||
**Stage 2 — Eval harness:**
|
||||
|
||||
```bash
|
||||
npm run eval:generation
|
||||
```
|
||||
|
||||
Fixture-based semantic checks. See `tools/eval/README.md`.
|
||||
|
||||
Both must pass before the task is complete.
|
||||
|
||||
---
|
||||
|
||||
## Adding a new entity
|
||||
|
||||
1. Add the entity's enums, DTOs, and `api` block to `domain/toir.api.dsl`.
|
||||
2. Run `npm run generate:api-summary`.
|
||||
3. **Before generating:** create `tools/eval/fixtures/<kebab>/meta.json`, `backend.assertions.json`, and `frontend.assertions.json` with expected patterns.
|
||||
4. Run `npm run eval:generation` — it will fail (entity files don't exist yet). That's expected.
|
||||
5. Generate backend and frontend artifacts (Steps 3–5).
|
||||
6. Run `npm run eval:generation` again — now it should pass.
|
||||
7. Run `node tools/validate-generation.mjs --artifacts-only` — both gates must pass.
|
||||
|
||||
> This is **eval-driven development**: write the failing eval first (step 3), then generate to make it pass (step 6). A passing eval confirms the LLM followed the rules.
|
||||
|
||||
---
|
||||
|
||||
## Changing an existing entity
|
||||
|
||||
**If the domain model changes** (new field, changed type, new FK, new enum):
|
||||
|
||||
1. Update `domain/toir.api.dsl` (enums, DTO attributes, `map` references).
|
||||
2. Run `npm run generate:api-summary`.
|
||||
3. Regenerate `server/prisma/schema.prisma`, run migration.
|
||||
4. Regenerate the affected modules and resources (Steps 4–6).
|
||||
|
||||
**If only the API contract changes** (different nullability, new endpoint, different HTTP method):
|
||||
|
||||
1. Update `domain/toir.api.dsl` only.
|
||||
2. Run `npm run generate:api-summary` to refresh `api-summary.json`.
|
||||
3. Run Steps 4–6. No Prisma migration needed.
|
||||
|
||||
---
|
||||
|
||||
## Artifact traceability matrix
|
||||
|
||||
| Artifact | DSL source | Prompt rule | Validator check |
|
||||
| ---------------------------------------------- | ---------------------- | ---------------------------------------------- | --------------------------------------- |
|
||||
| `server/prisma/schema.prisma` | `domain/*.api.dsl` | `prompts/prisma-rules.md` | `§validateBuildChecks` (file exists) |
|
||||
| `api-summary.json` | `domain/*.api.dsl` | — (deterministic) | `§validateBuildChecks` (freshness) |
|
||||
| `server/src/modules/<e>/*.ts` | `domain/*.api.dsl` | `prompts/backend-rules.md` | `§validateApiDslCoverage` |
|
||||
| `server/src/modules/<e>/dto/create-<e>.dto.ts` | `DTO.<E>Create` fields | `prompts/backend-rules.md §DTO-field-coverage` | `§validateApiDslCoverage` (field names) |
|
||||
| `server/src/modules/<e>/dto/update-<e>.dto.ts` | `DTO.<E>Update` fields | `prompts/backend-rules.md §DTO-field-coverage` | `§validateApiDslCoverage` (field names) |
|
||||
| `client/src/resources/<e>/<E>*.tsx` | `domain/*.api.dsl` | `prompts/frontend-rules.md` | `§validateApiDslCoverage` |
|
||||
| `client/src/auth/keycloak.ts` | — (Tier 4 handwritten) | `prompts/auth-rules.md` | `§validateAuthChecks` |
|
||||
| `toir-realm.json` | — (Tier 4 handwritten) | `prompts/auth-rules.md` | `§validateRealmChecks` |
|
||||
| `docker-compose.yml` | — (Tier 4 handwritten) | `prompts/runtime-rules.md` | `§validateRuntimeContractChecks` |
|
||||
|
||||
---
|
||||
|
||||
## Verification commands reference
|
||||
|
||||
| Command | When to use |
|
||||
| ----------------------------------------------------- | ------------------------------------------ |
|
||||
| `npm run generate:api-summary` | After any change to `domain/*.api.dsl` |
|
||||
| `npm run generate:openapi` | To regenerate OpenAPI 3.0.3 documentation |
|
||||
| `node tools/validate-generation.mjs --artifacts-only` | After every generation run (required) |
|
||||
| `node tools/validate-generation.mjs` | Before committing; requires installed deps |
|
||||
| `node tools/validate-generation.mjs --run-runtime` | End-to-end; requires running DB |
|
||||
|
||||
---
|
||||
|
||||
## OpenRouter invocation
|
||||
|
||||
Set environment variables before any LLM-mode tool call:
|
||||
|
||||
```
|
||||
OPENAI_API_KEY=<openrouter-key>
|
||||
OPENAI_BASE_URL=https://openrouter.ai/api/v1
|
||||
OPENAI_MODEL=<model-id>
|
||||
```
|
||||
|
||||
The `OPENAI_BASE_URL` variable is consumed by `tools/api-format-to-openapi/convert.mjs --mode llm`.
|
||||
For agent-driven generation via Cursor or direct API calls, the standard workflow above
|
||||
applies regardless of model provider.
|
||||
|
||||
---
|
||||
|
||||
## Auxiliary tools
|
||||
|
||||
### `tools/api-summary-to-openapi.mjs`
|
||||
|
||||
Generates `openapi.json` (OpenAPI 3.0.3) from `api-summary.json` deterministically.
|
||||
This is a documentation/integration artifact, not part of the core generation pipeline.
|
||||
|
||||
```bash
|
||||
npm run generate:openapi
|
||||
```
|
||||
|
||||
### `tools/api-format-to-openapi/`
|
||||
|
||||
Auxiliary integration tool for external consumers using the `api-format` JSON schema.
|
||||
Not connected to the main DSL pipeline. See `docs/AID_EXPORT_README.md` for details.
|
||||
Reference in New Issue
Block a user