diff --git a/CHANGELOG.md b/CHANGELOG.md index caaff16d..07d1fed3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Documented the `ESPERANTO_TTS_TIMEOUT` environment variable (default `300`s) in the environment reference; raise it for slow or self-hosted TTS providers so long podcast segments don't fail with a timeout (#937) - `SECURITY.md` with a coordinated-disclosure policy: how to privately report a vulnerability via GitHub's private vulnerability reporting, supported versions, and response expectations (#943) - LaTeX math rendering (KaTeX) now also applies to source content, source insights, Ask answers, transformation output, and the note editor preview — previously only chat had it (#269) +- Per-transformation model selection — each transformation can now be assigned its own language model from the transformation editor, overriding the global transformation default for that transformation only. Runs without an explicit model keep using the system default as before (#776) - "Refresh content" action on web-link sources — re-fetches the URL and re-embeds the source so its content stays current, available from the source card menu once processing has completed (translated across all 14 locales) (#259) ### Changed diff --git a/TRIAGE.md b/TRIAGE.md new file mode 100644 index 00000000..083722e4 --- /dev/null +++ b/TRIAGE.md @@ -0,0 +1,92 @@ +# Triage Rules — Open Notebook + +Project-specific triage rules. The `/triage` loop provides the **mechanism**; this file is the +**jurisprudence** for this repo. Keep it the single source of truth for how we label issues. + +## State funnel (outcome of triage) + +Every open issue lands in exactly **one** state. **Untriaged = no state label** — that's what +`/triage` scans for. + +| Label | Meaning | +|---|---| +| *(none)* | Not triaged yet. | +| `needs-vision` | Unsure if/how this fits — strategic call for the maintainers. | +| `needs-design` | Wanted, but the *how* isn't resolved — needs design/spec before it's ready. | +| `needs-info` | Waiting on the reporter to confirm or provide more information. | +| `ready` | Fully specified — the dev loop can pick it up. | +| **Close** | Use GitHub's native close reasons (duplicate / not planned). Link the canonical issue when duplicate/superseded. | + +## Type + +What kind of work it is. Apply one when it's clear. + +- `bug` · `enhancement` · `documentation` + +> `installation` is an **intake** label applied by the issue-creation workflow (not by triage). +> During triage, installation reports are routed to `area: deploy`. + +## Area — apply ALWAYS (one per issue) + +Which part of the system. This is the axis we use to route work and surface contributor-friendly +tasks, so every triaged issue should get one area when it clearly belongs to one. + +| Label | What goes here | +|---|---| +| `area: chat` | Conversation/chat, RAG retrieval, agentic responses, citations | +| `area: search` | Full-text and semantic search | +| `area: sources` | Source ingestion & processing (URLs, files, content extraction, chunking) | +| `area: notebooks` | Notebook features: notes, insights, transformations | +| `area: providers` | AI provider integrations + model configuration (OpenAI, Ollama, …) | +| `area: embeddings` | Embedding models, vectorization, semantic indexing | +| `area: podcast` | Podcast / audio generation | +| `area: database` | SurrealDB, persistence, schema, migrations | +| `area: ui` | Frontend (Next.js), UX, visual issues | +| `area: deploy` | Docker, deployment, k8s, reverse proxy, infra/setup | +| `area: offline` | Airgapped/offline operation (tokenizer caching, no network) | +| `area: i18n` | Internationalization / localization | + +## Bundling / epics + +- `umbrella` — a tracking issue grouping related work. +- `tracked-in-umbrella` — covered by an umbrella; follow the umbrella for progress. +- `bundled` — part of a thematic bundle. +- `upstream` — root cause lives in one of our libraries (below), not in this repo. + +### Consolidation: one issue vs. umbrella + +When several open issues circle the same topic, pick the model by **how decided the work is** — +not just by shared theme. The trap is filing many half-formed issues for a topic that isn't yet +decidable work; that inflates the backlog and reads as "90 things to do" when it's ~30 themes. + +- **Pre-vision / pre-design topic** → collapse into **one** issue (`needs-vision` or `needs-design`), + capture each request's signal (👍 counts, interested contributors) in its body, and **close the + rest as duplicates** pointing to it. A topic isn't N issues — it's one thinking space. +- **Already decomposed into real parallel tasks** → use `umbrella` + `tracked-in-umbrella`. Children + stay open because each is independently pickable (e.g. the multi-user umbrella #712). + +Rule of thumb: if the issues can't be worked until *we* make a call, they're one issue. If the call +is made and the work splits into things a contributor could pick up today, they're an umbrella with +children. **Never close an issue that has an active assignee/contributor or open PR** — link it as a +phase instead. + +## Ecosystem (our own libraries) + +Mark issues whose real home is an upstream lib: + +- `esperanto` — model abstraction layer (LLM / embedding / TTS / STT). +- `content-core` — content extraction. +- `podcast-creator` — podcast generation library. + +## Community + +- `good first issue` — small, well-scoped, newcomer-friendly. +- `help wanted` — we'd welcome a contributor to take this. + +## Rules + +- Assign **one state**, **one type**, and **one area** where each applies. Multiple + bundling/ecosystem labels are fine. +- `/triage` output must always carry a state (or be closed). Don't leave issues stateless. +- **Don't invent labels** — the set is curated. If something doesn't fit, raise it instead of adding a label. +- When closing as duplicate/superseded, link the canonical issue. diff --git a/api/models.py b/api/models.py index 05663d78..f5124606 100644 --- a/api/models.py +++ b/api/models.py @@ -112,6 +112,9 @@ class TransformationCreate(BaseModel): apply_default: bool = Field( False, description="Whether to apply this transformation by default" ) + model_id: Optional[str] = Field( + None, description="Model ID to use by default for this transformation" + ) class TransformationUpdate(BaseModel): @@ -126,6 +129,9 @@ class TransformationUpdate(BaseModel): apply_default: Optional[bool] = Field( None, description="Whether to apply this transformation by default" ) + model_id: Optional[str] = Field( + None, description="Model ID to use by default for this transformation" + ) class TransformationResponse(BaseModel): @@ -135,6 +141,7 @@ class TransformationResponse(BaseModel): description: str prompt: str apply_default: bool + model_id: Optional[str] = None created: str updated: str @@ -146,7 +153,9 @@ class TransformationExecuteRequest(BaseModel): ..., description="ID of the transformation to execute" ) input_text: str = Field(..., description="Text to transform") - model_id: str = Field(..., description="Model ID to use for the transformation") + model_id: Optional[str] = Field( + None, description="Model ID to use for this transformation run" + ) class TransformationExecuteResponse(BaseModel): @@ -154,7 +163,7 @@ class TransformationExecuteResponse(BaseModel): output: str = Field(..., description="Transformed text") transformation_id: str = Field(..., description="ID of the transformation used") - model_id: str = Field(..., description="Model ID used") + model_id: Optional[str] = Field(None, description="Model ID used") # Default Prompt API models diff --git a/api/routers/transformations.py b/api/routers/transformations.py index 1848ad54..50f61fd8 100644 --- a/api/routers/transformations.py +++ b/api/routers/transformations.py @@ -20,6 +20,20 @@ from open_notebook.graphs.transformation import graph as transformation_graph router = APIRouter() +def _transformation_response(transformation: Transformation) -> TransformationResponse: + return TransformationResponse( + id=transformation.id or "", + name=transformation.name, + title=transformation.title, + description=transformation.description, + prompt=transformation.prompt, + apply_default=transformation.apply_default, + model_id=transformation.model_id, + created=str(transformation.created), + updated=str(transformation.updated), + ) + + @router.get("/transformations", response_model=List[TransformationResponse]) async def get_transformations(): """Get all transformations.""" @@ -27,16 +41,7 @@ async def get_transformations(): transformations = await Transformation.get_all(order_by="name asc") return [ - TransformationResponse( - id=transformation.id or "", - name=transformation.name, - title=transformation.title, - description=transformation.description, - prompt=transformation.prompt, - apply_default=transformation.apply_default, - created=str(transformation.created), - updated=str(transformation.updated), - ) + _transformation_response(transformation) for transformation in transformations ] except Exception as e: @@ -50,25 +55,26 @@ async def get_transformations(): async def create_transformation(transformation_data: TransformationCreate): """Create a new transformation.""" try: + # Reject unknown model references up front (same check as execute); + # otherwise an invalid model_id is stored and only fails at run time. + if transformation_data.model_id: + model = await Model.get(transformation_data.model_id) + if not model: + raise HTTPException(status_code=404, detail="Model not found") + new_transformation = Transformation( name=transformation_data.name, title=transformation_data.title, description=transformation_data.description, prompt=transformation_data.prompt, apply_default=transformation_data.apply_default, + model_id=transformation_data.model_id, ) await new_transformation.save() - return TransformationResponse( - id=new_transformation.id or "", - name=new_transformation.name, - title=new_transformation.title, - description=new_transformation.description, - prompt=new_transformation.prompt, - apply_default=new_transformation.apply_default, - created=str(new_transformation.created), - updated=str(new_transformation.updated), - ) + return _transformation_response(new_transformation) + except HTTPException: + raise except InvalidInputError as e: raise HTTPException(status_code=400, detail=str(e)) except Exception as e: @@ -87,10 +93,14 @@ async def execute_transformation(execute_request: TransformationExecuteRequest): if not transformation: raise HTTPException(status_code=404, detail="Transformation not found") - # Validate model exists - model = await Model.get(execute_request.model_id) - if not model: - raise HTTPException(status_code=404, detail="Model not found") + model_id = execute_request.model_id or transformation.model_id + + # Validate explicit or transformation-specific model exists. + # None is allowed so the graph can use the configured transformation default. + if model_id: + model = await Model.get(model_id) + if not model: + raise HTTPException(status_code=404, detail="Model not found") # Execute the transformation result = await transformation_graph.ainvoke( @@ -98,13 +108,13 @@ async def execute_transformation(execute_request: TransformationExecuteRequest): input_text=execute_request.input_text, transformation=transformation, ), - config=dict(configurable={"model_id": execute_request.model_id}), + config=dict(configurable={"model_id": model_id}), ) return TransformationExecuteResponse( output=result["output"], transformation_id=execute_request.transformation_id, - model_id=execute_request.model_id, + model_id=model_id, ) except HTTPException: @@ -166,16 +176,7 @@ async def get_transformation(transformation_id: str): if not transformation: raise HTTPException(status_code=404, detail="Transformation not found") - return TransformationResponse( - id=transformation.id or "", - name=transformation.name, - title=transformation.title, - description=transformation.description, - prompt=transformation.prompt, - apply_default=transformation.apply_default, - created=str(transformation.created), - updated=str(transformation.updated), - ) + return _transformation_response(transformation) except HTTPException: raise except Exception as e: @@ -208,19 +209,17 @@ async def update_transformation( transformation.prompt = transformation_update.prompt if transformation_update.apply_default is not None: transformation.apply_default = transformation_update.apply_default + if "model_id" in transformation_update.model_fields_set: + # Validate a newly supplied model reference (allow clearing to None). + if transformation_update.model_id: + model = await Model.get(transformation_update.model_id) + if not model: + raise HTTPException(status_code=404, detail="Model not found") + transformation.model_id = transformation_update.model_id await transformation.save() - return TransformationResponse( - id=transformation.id or "", - name=transformation.name, - title=transformation.title, - description=transformation.description, - prompt=transformation.prompt, - apply_default=transformation.apply_default, - created=str(transformation.created), - updated=str(transformation.updated), - ) + return _transformation_response(transformation) except HTTPException: raise except InvalidInputError as e: diff --git a/api/transformations_service.py b/api/transformations_service.py index bc81c08c..bd0a0910 100644 --- a/api/transformations_service.py +++ b/api/transformations_service.py @@ -3,7 +3,7 @@ Transformations service layer using API. """ from datetime import datetime -from typing import Any, Dict, List, Union +from typing import Any, Dict, List, Optional, Union from loguru import logger @@ -17,39 +17,14 @@ class TransformationsService: def __init__(self): logger.info("Using API for transformations operations") - def get_all_transformations(self) -> List[Transformation]: - """Get all transformations.""" - transformations_data = api_client.get_transformations() - # Convert API response to Transformation objects - transformations = [] - for trans_data in transformations_data: - transformation = Transformation( - name=trans_data["name"], - title=trans_data["title"], - description=trans_data["description"], - prompt=trans_data["prompt"], - apply_default=trans_data["apply_default"], - ) - transformation.id = trans_data["id"] - transformation.created = datetime.fromisoformat( - trans_data["created"].replace("Z", "+00:00") - ) - transformation.updated = datetime.fromisoformat( - trans_data["updated"].replace("Z", "+00:00") - ) - transformations.append(transformation) - return transformations - - def get_transformation(self, transformation_id: str) -> Transformation: - """Get a specific transformation.""" - response = api_client.get_transformation(transformation_id) - trans_data = response if isinstance(response, dict) else response[0] + def _build_transformation(self, trans_data: Dict[str, Any]) -> Transformation: transformation = Transformation( name=trans_data["name"], title=trans_data["title"], description=trans_data["description"], prompt=trans_data["prompt"], apply_default=trans_data["apply_default"], + model_id=trans_data.get("model_id"), ) transformation.id = trans_data["id"] transformation.created = datetime.fromisoformat( @@ -60,6 +35,21 @@ class TransformationsService: ) return transformation + def get_all_transformations(self) -> List[Transformation]: + """Get all transformations.""" + transformations_data = api_client.get_transformations() + # Convert API response to Transformation objects + transformations = [] + for trans_data in transformations_data: + transformations.append(self._build_transformation(trans_data)) + return transformations + + def get_transformation(self, transformation_id: str) -> Transformation: + """Get a specific transformation.""" + response = api_client.get_transformation(transformation_id) + trans_data = response if isinstance(response, dict) else response[0] + return self._build_transformation(trans_data) + def create_transformation( self, name: str, @@ -77,21 +67,7 @@ class TransformationsService: apply_default=apply_default, ) trans_data = response if isinstance(response, dict) else response[0] - transformation = Transformation( - name=trans_data["name"], - title=trans_data["title"], - description=trans_data["description"], - prompt=trans_data["prompt"], - apply_default=trans_data["apply_default"], - ) - transformation.id = trans_data["id"] - transformation.created = datetime.fromisoformat( - trans_data["created"].replace("Z", "+00:00") - ) - transformation.updated = datetime.fromisoformat( - trans_data["updated"].replace("Z", "+00:00") - ) - return transformation + return self._build_transformation(trans_data) def update_transformation(self, transformation: Transformation) -> Transformation: """Update a transformation.""" @@ -104,6 +80,7 @@ class TransformationsService: "description": transformation.description, "prompt": transformation.prompt, "apply_default": transformation.apply_default, + "model_id": transformation.model_id, } response = api_client.update_transformation(transformation.id, **updates) trans_data = response if isinstance(response, dict) else response[0] @@ -114,6 +91,7 @@ class TransformationsService: transformation.description = trans_data["description"] transformation.prompt = trans_data["prompt"] transformation.apply_default = trans_data["apply_default"] + transformation.model_id = trans_data.get("model_id") transformation.updated = datetime.fromisoformat( trans_data["updated"].replace("Z", "+00:00") ) @@ -126,7 +104,7 @@ class TransformationsService: return True def execute_transformation( - self, transformation_id: str, input_text: str, model_id: str + self, transformation_id: str, input_text: str, model_id: Optional[str] = None ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]: """Execute a transformation on input text.""" result = api_client.execute_transformation( diff --git a/frontend/src/app/(dashboard)/transformations/components/TransformationEditorDialog.tsx b/frontend/src/app/(dashboard)/transformations/components/TransformationEditorDialog.tsx index 4143270a..34c3b180 100644 --- a/frontend/src/app/(dashboard)/transformations/components/TransformationEditorDialog.tsx +++ b/frontend/src/app/(dashboard)/transformations/components/TransformationEditorDialog.tsx @@ -1,6 +1,6 @@ 'use client' -import { useEffect, useId } from 'react' +import { useEffect, useId, useMemo } from 'react' import { Controller, useForm } from 'react-hook-form' import { zodResolver } from '@hookform/resolvers/zod' import { z } from 'zod' @@ -11,18 +11,33 @@ import { Textarea } from '@/components/ui/textarea' import { Checkbox } from '@/components/ui/checkbox' import { Label } from '@/components/ui/label' import { MarkdownEditor } from '@/components/ui/markdown-editor' -import { useCreateTransformation, useUpdateTransformation, useTransformation } from '@/lib/hooks/use-transformations' +import { + Select, + SelectContent, + SelectItem, + SelectTrigger, + SelectValue, +} from '@/components/ui/select' +import { + useCreateTransformation, + useUpdateTransformation, + useTransformation, +} from '@/lib/hooks/use-transformations' +import { useModels } from '@/lib/hooks/use-models' import { Transformation } from '@/lib/types/transformations' import { useQueryClient } from '@tanstack/react-query' import { TRANSFORMATION_QUERY_KEYS } from '@/lib/hooks/use-transformations' import { useTranslation } from '@/lib/hooks/use-translation' +const DEFAULT_MODEL_VALUE = '__default_transformation_model__' + const transformationSchema = z.object({ name: z.string().min(1), title: z.string().min(1), description: z.string().optional(), prompt: z.string().min(1), apply_default: z.boolean().optional(), + model_id: z.string().nullable().optional(), }) type TransformationFormData = z.infer @@ -40,10 +55,16 @@ export function TransformationEditorDialog({ open, onOpenChange, transformation const defaultId = useId() const descriptionId = useId() const promptId = useId() + const modelId = useId() const isEditing = Boolean(transformation) const { data: fetchedTransformation, isLoading } = useTransformation(transformation?.id ?? '', { enabled: open && Boolean(transformation?.id), }) + const { data: models = [], isLoading: isLoadingModels } = useModels() + const languageModels = useMemo( + () => models.filter((model) => model.type === 'language'), + [models] + ) const createTransformation = useCreateTransformation() const updateTransformation = useUpdateTransformation() const queryClient = useQueryClient() @@ -61,12 +82,20 @@ export function TransformationEditorDialog({ open, onOpenChange, transformation description: '', prompt: '', apply_default: false, + model_id: null, }, }) useEffect(() => { if (!open) { - reset({ name: '', title: '', description: '', prompt: '', apply_default: false }) + reset({ + name: '', + title: '', + description: '', + prompt: '', + apply_default: false, + model_id: null, + }) return } @@ -77,6 +106,7 @@ export function TransformationEditorDialog({ open, onOpenChange, transformation description: source?.description ?? '', prompt: source?.prompt ?? '', apply_default: source?.apply_default ?? false, + model_id: source?.model_id ?? null, }) }, [open, transformation, fetchedTransformation, reset]) @@ -90,6 +120,7 @@ export function TransformationEditorDialog({ open, onOpenChange, transformation description: data.description || undefined, prompt: data.prompt, apply_default: Boolean(data.apply_default), + model_id: data.model_id || null, }, }) queryClient.invalidateQueries({ queryKey: TRANSFORMATION_QUERY_KEYS.transformation(transformation.id) }) @@ -100,6 +131,7 @@ export function TransformationEditorDialog({ open, onOpenChange, transformation description: data.description || '', prompt: data.prompt, apply_default: Boolean(data.apply_default), + model_id: data.model_id || null, }) } @@ -170,24 +202,60 @@ export function TransformationEditorDialog({ open, onOpenChange, transformation )} /> -
+
+ ( - field.onChange(Boolean(checked))} - /> + )} /> -
+
+ ( + field.onChange(Boolean(checked))} + /> + )} + /> + +
+