Scriberr/devnotes/engine-worker-sprint-tracker.md

# Sprint Run Tracker: Engine Worker Integration

Run ID: `EWI`

Status: completed through EWI-Sprint 10. Docker/deployment packaging updates remain deferred by request.

This tracker belongs to `devnotes/engine-worker-sprints.md` and the implementation spec in `devnotes/engine-worker-integration-spec.md`.

## EWI-Sprint 0: Integration Inventory and Commit Plan

Status: completed

Completed tasks:

- Inventoried server startup, config, schema, repository, queue, transcription stack, API placeholders, docs, Docker, and test fixtures.
- Documented the legacy adapter deletion targets.
- Documented API/service seams for create, submit, retry, cancel, transcript, events, logs, executions, models, and queue stats.
- Added structured logging requirements for config, provider, worker, queue, orchestration, and terminal states.
- Added a sprint-by-sprint commit plan for EWI-Sprints 1-10.

Artifacts:

- `devnotes/engine-worker-sprint-0-inventory.md`

Verification:

- Inventory-only sprint. No runtime code changed.
- Focused repository inspection completed with `rg`, `find`, and targeted source reads.

## EWI-Sprint 1: Config and Engine Module Wiring

Status: completed

Completed tasks:

- Added local engine module wiring with `require scriberr-engine v0.0.0` and `replace scriberr-engine => ./references/engine`.
- Added `config.EngineConfig` and `config.WorkerConfig`.
- Added `config.LoadWithError()` for startup-failing validation while retaining `config.Load()` for compatibility.
- Parsed and validated all `SPEECH_ENGINE_*` and `TRANSCRIPTION_*` env vars from the spec.
- Updated server startup to fail clearly on invalid config.
- Added structured startup logging for engine and worker configuration.
- Added focused config tests before implementation.

Artifacts:

- `go.mod`
- `cmd/server/main.go`
- `internal/config/config.go`
- `internal/config/config_test.go`

Verification:

- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/config` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go vet ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `git diff --check` passed.

## EWI-Sprint 2: Engine Provider Abstraction

Status: completed

Completed tasks:

- Added `internal/transcription/engineprovider` provider and registry interfaces.
- Added internal provider request/result/capability types so `scriberr-engine` types do not leak outside the provider boundary.
- Added static provider registry with deterministic capability aggregation.
- Added local provider wrapper for `scriberr-engine/speech/engine`.
- Mapped Scriberr transcription and diarization requests to local engine requests.
- Forced token timestamps for local transcription requests.
- Mapped engine words and diarization segments to public-safe internal result structs.
- Added model capability discovery from the engine model specs with install state through `IsModelInstalled`.
- Added provider error sanitization for paths and token-like values.
- Added focused fake-engine tests for mapping, empty words, capabilities, diarization speakers, close behavior, and sanitized errors.
- Updated the main module to `go 1.26` because the local `scriberr-engine` module declares `go 1.26`.

Artifacts:

- `internal/transcription/engineprovider/types.go`
- `internal/transcription/engineprovider/registry.go`
- `internal/transcription/engineprovider/local_provider.go`
- `internal/transcription/engineprovider/sanitize.go`
- `internal/transcription/engineprovider/*_test.go`
- `go.mod`
- `go.sum`

Verification:

- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/engineprovider` passed.
- `GOCACHE=/tmp/scriberr-go-cache go vet ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed with escalation because an existing webhook integration test opens a local `httptest` listener.
- `git diff --check` passed.
- Verified no non-provider Go package imports `scriberr-engine`.

## EWI-Sprint 3: Queue Schema and Repository Methods

Status: completed

Completed tasks:

- Added durable queue/lease/progress fields to `models.TranscriptionJob`.
- Added queue claim and claim-expiry indexes to the target schema.
- Extended `JobRepository` with durable worker methods for enqueue, FIFO claim, lease renewal, startup recovery, progress, completion, failure, cancellation, and execution listing.
- Implemented transactional terminal updates that keep the job row and latest execution row consistent.
- Added focused repository tests for schema/indexes, enqueue, FIFO claim, concurrent claim deduplication, owner-only lease renewal, orphan recovery, progress updates, terminal transitions, and execution listing.
- Updated existing legacy transcription test mocks to satisfy the expanded repository interface until the legacy stack is removed in later sprints.

Artifacts:

- `internal/models/transcription.go`
- `internal/database/schema.go`
- `internal/repository/implementations.go`
- `internal/repository/job_queue_test.go`
- `internal/transcription/adapters_test.go`
- `tests/test_helpers.go`

Verification:

- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/repository -run 'TestJobRepository'` passed.
- `GOCACHE=/tmp/scriberr-go-cache go vet ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed with escalation because an existing webhook integration test opens a local `httptest` listener.
- `git diff --check` passed.

## EWI-Sprint 4: Durable Worker Service

Status: completed

Completed tasks:

- Added `internal/transcription/worker` with the public queue service interface from the sprint plan.
- Implemented durable enqueue plus non-blocking worker wake signaling.
- Implemented worker startup recovery through `RecoverOrphanedProcessing`.
- Implemented polling/claim loop with configurable worker count, poll interval, lease timeout, renew interval, and stop timeout.
- Implemented lease renewal while processors are running.
- Implemented process-local cancel tracking for running jobs.
- Implemented cancel behavior for queued jobs, process-local running jobs, orphaned processing jobs, and terminal-state conflicts.
- Implemented user-scoped queue stats with process-local running counts.
- Added structured lifecycle, enqueue, worker, lease-renewal, cancellation, and shutdown logs.
- Added focused worker tests with fake processors for enqueue/wake/complete, cancel queued, cancel running, lease renewal, stop cancellation, stats, and cancel conflicts.
- Added repository status-count support needed by worker stats.

Artifacts:

- `internal/transcription/worker/service.go`
- `internal/transcription/worker/service_test.go`
- `internal/repository/implementations.go`
- `internal/transcription/adapters_test.go`
- `tests/test_helpers.go`

Verification:

- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/worker` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./tests -run '^$'` passed.
- `GOCACHE=/tmp/scriberr-go-cache go vet ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed with escalation because an existing webhook integration test opens a local `httptest` listener.
- `git diff --check` passed.

## EWI-Sprint 5: Orchestrator, Transcript Mapping, and Speaker Merge

Status: completed

Completed tasks:

- Added `internal/transcription/orchestrator` with a worker-compatible processor.
- Added canonical transcript structs, JSON parsing, mapper, fallback segment generation, and legacy plain-text/older-JSON fallback parsing.
- Implemented overlap-based speaker assignment for words and segments with stable public `SPEAKER_00` labels.
- Implemented provider/model/language/task/diarization request resolution.
- Created execution rows at processor start with sanitized request/config metadata.
- Published progress stages for preparing, transcribing, diarizing, merging, saving, completed, failed, and canceled paths.
- Wrote canonical transcript JSON to the configured transcript output directory and returned the internal output path for worker completion.
- Preserved `words: []` when token timestamps are absent.
- Sanitized provider failures to redact paths and token-like values.
- Distinguished context cancellation from provider failure.

Artifacts:

- `internal/transcription/orchestrator/processor.go`
- `internal/transcription/orchestrator/transcript.go`
- `internal/transcription/orchestrator/processor_test.go`
- `internal/transcription/orchestrator/transcript_test.go`

Verification:

- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/...` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go vet ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `git diff --check` passed.

## EWI-Sprint 6: API Wiring for Real Queue Execution

Status: completed

Completed tasks:

- Added API handler injection for durable queue service and engine provider registry.
- Wired create, submit, and retry to enqueue through the queue service.
- Mapped queue shutdown to `503 SERVICE_UNAVAILABLE` without deleting durable job rows.
- Wired cancel to queue service cancellation and mapped terminal-state conflicts to `409`.
- Added progress fields to transcription get/list responses.
- Implemented canonical transcript endpoint parsing for JSON, legacy text, and older JSON without `words`.
- Implemented executions endpoint with sanitized execution metadata and processing duration.
- Implemented logs endpoint as authenticated plain text derived from execution metadata/log files with path/token redaction.
- Implemented model listing from provider capabilities with installed/default flags.
- Updated queue stats to use queue service stats when injected, including canceled/running counts.
- Added an API event publisher adapter for orchestrator progress events with path-safe payloads.
- Added focused API tests for queue-backed create/retry/cancel, queue unavailable errors, transcript/execution/log/model/stats responses, and leak-safe errors.

Artifacts:

- `internal/api/router.go`
- `internal/api/transcription_handlers.go`
- `internal/api/admin_handlers.go`
- `internal/api/response_models.go`
- `internal/api/engine_worker_api_test.go`
- API test helper updates

Verification:

- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/...` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go vet ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `git diff --check` passed.

## EWI-Sprint 7: Server Startup, Shutdown, and Legacy Adapter Removal

Status: completed

Completed tasks:

- Replaced server startup wiring with the local engine provider registry, orchestrator processor, and durable worker service.
- Removed server startup dependencies on legacy `internal/queue`, Python adapter registration, unified processor, quick transcription, and embedded Python environment bootstrap.
- Started durable transcription workers after database/repository/provider/API construction so worker startup recovery runs before claims.
- Wired API handler to the queue service and provider registry from real server startup.
- Updated shutdown to stop HTTP serving, stop workers, close the local provider, and close the database.
- Added server regression coverage proving `cmd/server/main.go` no longer references legacy Python startup symbols.
- Added worker coverage for recovering orphaned processing jobs before workers claim work.
- Stopped compiling the legacy Python adapter stack by placing adapters, registry, pipeline, unified service, quick transcription, and obsolete adapter/webhook tests behind a `legacy_python` build tag.
- Added package stubs for legacy-tagged packages so normal package discovery stays clean.

Artifacts:

- `cmd/server/main.go`
- `cmd/server/main_test.go`
- `internal/transcription/worker/service_test.go`
- `internal/transcription/doc.go`
- `internal/transcription/adapters/doc.go`
- `internal/transcription/registry/doc.go`
- `internal/transcription/pipeline/doc.go`
- legacy Python adapter files tagged with `legacy_python`

Verification:

- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/...` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go vet ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./tests -run '^$'` passed.
- `git diff --check` passed.

## EWI-Sprint 8: Real Engine Integration Tests and Performance Smoke

Status: completed

Completed tasks:

- Added opt-in real local engine integration tests gated by `SCRIBERR_ENGINE_ITEST=1`.
- Added `test-audio/jfk.wav` real transcription smoke coverage with non-empty text, non-nil words, provider identity, timing logs, and path-leak assertions.
- Added auto-download-disabled missing-model coverage that uses an isolated empty cache and asserts sanitized model-unavailable behavior without downloads.
- Added a one-iteration-friendly benchmark for local JFK timing without brittle pass/fail thresholds.
- Added clean skip handling for disabled opt-in flag, missing `ffmpeg`, missing fixture audio, unavailable runtime libraries, CUDA/runtime issues, and network/download failures.
- Added concise smoke notes with commands for JFK transcription, optional cache override, and benchmark/manual performance recording.

Artifacts:

- `internal/transcription/engineprovider/real_engine_integration_test.go`
- `devnotes/engine-worker-sprint-8-smoke.md`

Verification:

- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/...` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go vet ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `SCRIBERR_ENGINE_ITEST=1 GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/engineprovider -run TestRealEngineAutoDownloadDisabledMissingModelIsSanitized -v` passed.
- `git diff --check` passed.

## EWI-Sprint 9: Hardening, Cleanup

Status: completed

Completed tasks:

- Ran the full backend test/vet baseline.
- Ran focused race checks for repository queue claiming and worker recovery/cancellation paths.
- Ran the opt-in `jfk.wav` real engine smoke path; the test passed with a documented external DNS/model-download skip.
- Removed stale transcription package architecture docs and replaced them with the active engine provider, orchestrator, and worker flow.
- Deferred Docker, compose, and deployment documentation updates by request.

Artifacts:

- `internal/transcription/README.md`
- `devnotes/engine-worker-sprint-tracker.md`

Verification:

- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/...` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go vet ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test -race ./internal/repository -run TestJobRepositoryConcurrentClaimsDoNotDuplicateJobs` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test -race ./internal/transcription/worker -run 'TestService(EnqueueWakeAndComplete|StartRecoversOrphanedProcessingBeforeWorkersClaim|CancelRunning)'` passed.
- `SCRIBERR_ENGINE_ITEST=1 GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/engineprovider -run TestRealEngineJFKTranscription -v` passed with a skip because external model download DNS was unavailable.
- `git diff --check` passed.

## EWI-Sprint 10: Hardening, Cleanup, and Release Candidate

Status: completed

Completed tasks:

- Reviewed the branch implementation against the engine worker integration spec and sprint acceptance criteria.
- Reconciled enqueue failure behavior with the durable queue contract: `503 SERVICE_UNAVAILABLE` responses keep the queued job durable for later recovery.
- Applied default and selected transcription profiles to create/submit jobs, with request options overriding profile values.
- Removed duplicate global SSE progress events while preserving job-specific and global delivery through the broker.
- Sanitized `last_error` in public transcription responses, matching executions/logs redaction.
- Preserved log endpoint line breaks while redacting paths and token-like values.
- Hardened the local provider against nil engine transcription/diarization results.
- Confirmed `scriberr-engine` imports remain isolated to the provider package and opt-in real-engine tests.
- Confirmed Docker/deployment work is intentionally not part of this release-candidate pass.

Artifacts:

- `internal/api/admin_handlers.go`
- `internal/api/engine_worker_api_test.go`
- `internal/api/events_test.go`
- `internal/api/response_models.go`
- `internal/api/router.go`
- `internal/api/transcription_handlers.go`
- `internal/api/transcriptions_test.go`
- `internal/api/types.go`
- `internal/transcription/engineprovider/local_provider.go`
- `internal/transcription/engineprovider/local_provider_test.go`
- `devnotes/engine-worker-sprint-tracker.md`

Verification:

- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api -run 'Test(CreateReturnsServiceUnavailableWhenQueueStopped|RetryPreservesNewJobWhenQueueStopped|TranscriptionCreateAppliesDefaultAndSelectedProfiles|GlobalSSEReceivesTranscriptionProgressOnce)'` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/engineprovider -run 'TestLocalProvider(TranscribeRejectsNilEngineResult|DiarizeRejectsNilEngineResult)'` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go vet ./internal/api ./internal/config ./internal/database ./internal/repository ./internal/transcription/... ./cmd/server ./pkg/logger ./pkg/middleware` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test -race ./internal/repository -run TestJobRepositoryConcurrentClaimsDoNotDuplicateJobs` passed.
- `GOCACHE=/tmp/scriberr-go-cache go test -race ./internal/transcription/worker -run 'TestService(EnqueueWakeAndComplete|StartRecoversOrphanedProcessingBeforeWorkersClaim|CancelRunning)'` passed.
- `SCRIBERR_ENGINE_ITEST=1 GOCACHE=/tmp/scriberr-go-cache go test ./internal/transcription/engineprovider -run TestRealEngineJFKTranscription -v` passed with a skip because external model download DNS was unavailable.
- `git diff --check` passed.

Additional sprint assessment:

- No additional runtime implementation sprint is required for the current engine worker integration spec.
- A separate deployment/packaging sprint is still needed before changing Dockerfiles, compose files, release packaging, or deployment docs.