Updates 1
CI / ci (push) Canceled after 0s

The session built:

Appointments page: a single list, newest first.
Block Time: edit and delete.
Customer CSV export: with barber notes excluded.
Guest-link regression test: a test that an appointment id alone is rejected, plus fixes to the stale docs.

It built the scheduler, job runner with live progress, backup and restore, and the health alerts. Its own report says a real delete-and-restore on throwaway Docker passed, but I haven’t repeated that. It also says real SMTP and large dumps were not exercised. The manager image needs rebuilding for the backups to work, because it now installs 7zip.
This commit is contained in:
Stefan Lazic committed 2026-10-17 22:29:00 +01:00
1 parent 9f981173f4
commit 17eee5b932
95 files changed
+7955 -150

No files matched your search

+16
View File
@@ -42,3 +42,19 @@ STACKS_DIR="/opt/nextchair/stacks"
# "Update" button (or `docker compose up -d --force-recreate`) to pick it
# up.
NEXTCHAIR_IMAGE="nextchair-app:latest"
# ── Scheduler, backups, alerts (all optional) ───────────────────────────
# The built-in scheduler (notification cron, daily Google reviews sync,
# health checks, scheduled backups) starts with the Manager. Set to 1 to
# turn it off, e.g. to keep using the host-crontab fallback
# (stacks/infra/scripts/process-cron.sh) instead — never run both.
# MANAGER_SCHEDULER_DISABLED="1"
# Scheduled-backup times and the dashboard's dates use the container's
# timezone (UTC by default). Set e.g. TZ="Europe/London" to change it.
# TZ="Europe/London"
# Largest .7z accepted by the Restore page's upload (bytes; default 4 GiB).
# BACKUP_UPLOAD_MAX_BYTES="4294967296"
# An external dead-man's-switch (healthchecks.io, Better Stack…): the
# scheduler GETs this URL every 5 minutes, so you're told if the whole
# VPS or the Manager is down — something a self-hosted check can't report.
# MANAGER_HEARTBEAT_URL="https://hc-ping.com/your-uuid"
+5
View File
@@ -30,6 +30,11 @@ ENV NODE_ENV=production
# stacks/infra/compose.yaml's manager service (docker.sock).
RUN apk add --no-cache docker-cli docker-cli-compose
# 7-Zip for encrypted backups (`7zz a -mhe=on -p…`, see src/lib/sevenzip.ts).
# Alpine's package is named `7zip` and installs /usr/bin/7zz (checked
# against node:22-alpine, Alpine 3.24: `apk add 7zip` → 7-Zip 26.01).
RUN apk add --no-cache 7zip
RUN addgroup --system --gid 1001 nodejs && adduser --system --uid 1001 manager
COPY --from=builder --chown=manager:nodejs /app/node_modules ./node_modules
+49 -2
View File
@@ -30,12 +30,59 @@ copied and adapted, not imported — see `src/lib/auth.ts` and
reloads Caddy, starts the container, and runs its migrations. See
`src/lib/provisioning.ts` for the exact sequence.
- Per business: start / stop / restart / update (recreate the container
against the current shared image tag — not a source rebuild) / view
recent logs.
against the current shared image tag — not a source rebuild, and run as
a background job with live progress and an optional "back up first") /
view recent logs.
- Does **not** duplicate the `/install` wizard — after provisioning, the
operator is linked to `https://<domain>/install` to finish that
business's own setup.
## Scheduler, jobs, backups and alerts
All of this runs inside the Manager process — no host crontab, no extra
service. State lives next to `global-settings.json`, under
`STACKS_DIR/manager/` (and `STACKS_DIR/backups/`), so it survives rebuilds.
- **Scheduler** (`src/instrumentation.ts` → `src/lib/scheduler*.ts`): one
guarded interval, a task registry with per-task intervals and
per-business overlap guards, last-run results in `scheduler-state.json`
(shown at Settings → Monitoring & scheduler). Tasks: notification cron
(`POST /api/cron/process-notifications` with each stack's `CRON_SECRET`,
every 30 s by default), daily Google reviews sync, health checks (every
3 min by default), scheduled backups, hourly housekeeping, and an
optional external heartbeat (`MANAGER_HEARTBEAT_URL`). Turn it off with
`MANAGER_SCHEDULER_DISABLED=1`.
**Fallback:** `stacks/infra/scripts/process-cron.sh` (host crontab) still
works and does the same notification/reviews calls — use it only if the
scheduler is disabled, never both.
- **Jobs** (`src/lib/jobs.ts`, `locks.ts`): Update, Backup, Restore and
Delete run as background jobs writing `manager/jobs/<id>.json` (status,
steps, log tail); the UI polls `GET /api/jobs/<id>`. Each holds a
per-business lock (`manager/locks/<slug>.json`) so they can't overlap.
A lock whose heartbeat is older than 2 minutes is stale and may be taken
over; on boot, jobs left "running" are marked failed and their locks
released.
- **Backups** (`src/lib/backup*.ts`, `restore.ts`; UI at a business's
*Backups* page and the global *Restore* page): an encrypted `.7z`
(`7zz`, AES-256, encrypted headers) of the database dump, media and
config, in `STACKS_DIR/backups/<slug>/`. Written as `.partial`, verified
with `7zz t`, then renamed; the newest N (default 2, per business) are
kept. Safety backups (before an update or a replacing restore) and final
backups (before a delete) are kept separately and never evict them. One
passphrase protects everything: generated at the first enable / first
*Back up now*, stored encrypted (`MANAGER_SETTINGS_ENCRYPTION_KEY`),
**shown once** — save it, a fresh server needs it to restore. Restore
works onto an existing business (after a safety backup) or a server that
never had it; see `RESTORE_STEPS` in `src/lib/restore.ts`. A backup
whose last migration is newer than the app image's is refused.
- **Alerts** (`src/lib/health-watcher.ts`): per-business ok / degraded /
down. Two consecutive bad checks before the first email; emails on
change and on recovery, and a daily reminder while still bad. Set the
operator address and toggles at Settings → Monitoring & scheduler (uses
the Email settings' SMTP). Businesses stopped from the Manager, or mid
update/backup/restore, are not probed. An external check is still needed
for "the whole VPS is down" — use `MANAGER_HEARTBEAT_URL`.
## How auth works
A single dedicated platform-operator credential (`MANAGER_EMAIL` /
+32 -1
View File
@@ -1,7 +1,7 @@
import { NextRequest } from "next/server";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { createSessionToken, SESSION_COOKIE_NAME } from "@/lib/session";
import proxy from "../proxy";
import proxy, { config } from "../proxy";
/**
* The installation-token panel (and its regenerate action) lives under
@@ -64,3 +64,34 @@ describe("proxy — Manager auth gate", () => {
expect(response.status).toBe(200);
});
});
describe("proxy — API routes", () => {
const originalSecret = process.env.MANAGER_SESSION_SECRET;
beforeEach(() => {
process.env.MANAGER_SESSION_SECRET = "test-session-secret-value";
});
afterEach(() => {
if (originalSecret === undefined) delete process.env.MANAGER_SESSION_SECRET;
else process.env.MANAGER_SESSION_SECRET = originalSecret;
});
it.each(["/api/jobs/abc123-deadbeef", "/api/backups/acme/acme__20261017T030000Z__regular.7z"])("gates %s behind the login", (pathname) => {
const response = proxy(new NextRequest(new URL(`http://localhost${pathname}`)));
expect(response.status).toBe(307);
expect(response.headers.get("location")).toBe("http://localhost/login");
});
it("lets a signed-in request through to a job poll", () => {
const token = createSessionToken("op@example.com");
const response = proxy(new NextRequest(new URL("http://localhost/api/jobs/abc123-deadbeef"), { headers: { cookie: `${SESSION_COOKIE_NAME}=${token}` } }));
expect(response.status).toBe(200);
});
it("excludes ONLY the upload route from the proxy (it would truncate large bodies; the route checks the session itself)", () => {
const matcher = new RegExp(`^${config.matcher[0]!}$`);
expect(matcher.test("/api/backups/upload")).toBe(false);
expect(matcher.test("/api/backups/acme/file.7z")).toBe(true);
expect(matcher.test("/api/jobs/x")).toBe(true);
expect(matcher.test("/dashboard")).toBe(true);
});
});
@@ -0,0 +1,114 @@
import { mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { NextRequest } from "next/server";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { createSessionToken, SESSION_COOKIE_NAME } from "@/lib/session";
import { GET as downloadBackup } from "../backups/[slug]/[file]/route";
import { POST as uploadBackup } from "../backups/upload/route";
import { GET as getJob } from "../jobs/[id]/route";
describe("manager API routes", () => {
let stacksDir: string;
const saved = { stacks: process.env.STACKS_DIR, secret: process.env.MANAGER_SESSION_SECRET, max: process.env.BACKUP_UPLOAD_MAX_BYTES };
beforeEach(async () => {
stacksDir = await mkdtemp(path.join(tmpdir(), "nc-manager-routes-"));
process.env.STACKS_DIR = stacksDir;
process.env.MANAGER_SESSION_SECRET = "test-session-secret-value";
delete process.env.BACKUP_UPLOAD_MAX_BYTES;
});
afterEach(async () => {
for (const [key, value] of [["STACKS_DIR", saved.stacks], ["MANAGER_SESSION_SECRET", saved.secret], ["BACKUP_UPLOAD_MAX_BYTES", saved.max]] as const) {
if (value === undefined) delete process.env[key];
else process.env[key] = value;
}
await rm(stacksDir, { recursive: true, force: true });
});
const cookie = () => `${SESSION_COOKIE_NAME}=${createSessionToken("op@example.com")}`;
function uploadRequest(body: Uint8Array | null, headers: Record<string, string> = {}) {
return new NextRequest("http://localhost/api/backups/upload", {
method: "POST",
headers: { "content-type": "application/octet-stream", ...headers },
body: body ? new ReadableStream({ start(c) { c.enqueue(body); c.close(); } }) : null,
// undici requires this for streaming request bodies
duplex: "half",
});
}
describe("POST /api/backups/upload", () => {
it("rejects an unauthenticated upload without reading it (the route is outside the proxy, so it must check itself)", async () => {
const response = await uploadBackup(uploadRequest(new Uint8Array([1, 2, 3])));
expect(response.status).toBe(401);
await expect(readdir(path.join(stacksDir, "backups", "_uploads"))).rejects.toThrow();
});
it("streams the body to backups/_uploads/<id>.7z and returns the id and size", async () => {
const payload = new Uint8Array(5000).fill(7);
const response = await uploadBackup(uploadRequest(payload, { cookie: cookie() }));
expect(response.status).toBe(200);
const body = (await response.json()) as { uploadId: string; sizeBytes: number };
expect(body.uploadId).toMatch(/^[a-f0-9]{16}$/);
expect(body.sizeBytes).toBe(5000);
expect((await readFile(path.join(stacksDir, "backups", "_uploads", `${body.uploadId}.7z`))).length).toBe(5000);
});
it("enforces the size cap from Content-Length before reading anything", async () => {
process.env.BACKUP_UPLOAD_MAX_BYTES = "1000";
const response = await uploadBackup(uploadRequest(new Uint8Array(10), { cookie: cookie(), "content-length": "5000" }));
expect(response.status).toBe(413);
});
it("enforces the cap while streaming too (a client can understate Content-Length) and deletes the partial file", async () => {
process.env.BACKUP_UPLOAD_MAX_BYTES = "1000";
const response = await uploadBackup(uploadRequest(new Uint8Array(5000), { cookie: cookie() }));
expect(response.status).toBe(413);
expect(await readdir(path.join(stacksDir, "backups", "_uploads"))).toEqual([]);
});
it("rejects an empty upload", async () => {
const response = await uploadBackup(uploadRequest(new Uint8Array(0), { cookie: cookie() }));
expect(response.status).toBe(400);
});
});
describe("GET /api/backups/[slug]/[file]", () => {
it("streams a stored archive as an attachment", async () => {
const name = "acme__20261017T030000Z__regular.7z";
await mkdir(path.join(stacksDir, "backups", "acme"), { recursive: true });
await writeFile(path.join(stacksDir, "backups", "acme", name), "ARCHIVE-BYTES");
const response = await downloadBackup(new Request("http://localhost/x"), { params: Promise.resolve({ slug: "acme", file: name }) });
expect(response.status).toBe(200);
expect(response.headers.get("content-disposition")).toBe(`attachment; filename="${name}"`);
expect(response.headers.get("content-length")).toBe("13");
expect(await response.text()).toBe("ARCHIVE-BYTES");
});
it.each([
["acme", "../../../etc/passwd"],
["acme", "other__20261017T030000Z__regular.7z"],
["../acme", "acme__20261017T030000Z__regular.7z"],
["acme", "acme__20261017T030000Z__regular.7z"], // well-formed but absent
])("404s for %s / %s", async (slug, file) => {
const response = await downloadBackup(new Request("http://localhost/x"), { params: Promise.resolve({ slug, file }) });
expect(response.status).toBe(404);
});
});
describe("GET /api/jobs/[id]", () => {
it("returns the job record, and 404 for unknown or malformed ids", async () => {
const id = "abc123-deadbeef";
await mkdir(path.join(stacksDir, "manager", "jobs"), { recursive: true });
await writeFile(path.join(stacksDir, "manager", "jobs", `${id}.json`), JSON.stringify({ id, status: "running", steps: [], logTail: [] }));
const ok = await getJob(new Request("http://localhost/x"), { params: Promise.resolve({ id }) });
expect(ok.status).toBe(200);
expect(((await ok.json()) as { status: string }).status).toBe("running");
expect((await getJob(new Request("http://localhost/x"), { params: Promise.resolve({ id: "nope" }) })).status).toBe(404);
expect((await getJob(new Request("http://localhost/x"), { params: Promise.resolve({ id: "../../global-settings" }) })).status).toBe(404);
});
});
});
@@ -0,0 +1,36 @@
import { createReadStream } from "node:fs";
import { stat } from "node:fs/promises";
import { Readable } from "node:stream";
import { NextResponse } from "next/server";
import { backupFilePath } from "@/lib/backup-store";
export const dynamic = "force-dynamic";
/** Streams a stored archive as a download — never buffered in memory, since archives can be large. The .7z is encrypted; the download itself exposes nothing without the passphrase. */
export async function GET(_request: Request, { params }: { params: Promise<{ slug: string; file: string }> }) {
const { slug, file } = await params;
let filePath: string;
try {
filePath = backupFilePath(slug, file);
} catch {
return NextResponse.json({ error: "Not found." }, { status: 404 });
}
let size: number;
try {
size = (await stat(filePath)).size;
} catch {
return NextResponse.json({ error: "Not found." }, { status: 404 });
}
const body = Readable.toWeb(createReadStream(filePath)) as ReadableStream<Uint8Array>;
return new Response(body, {
headers: {
"Content-Type": "application/x-7z-compressed",
"Content-Length": String(size),
"Content-Disposition": `attachment; filename="${file}"`,
"Cache-Control": "no-store",
},
});
}
@@ -0,0 +1,71 @@
import { createWriteStream } from "node:fs";
import { mkdir, rm } from "node:fs/promises";
import path from "node:path";
import { Readable, Transform } from "node:stream";
import { pipeline } from "node:stream/promises";
import { NextResponse, type NextRequest } from "next/server";
import { uploadsDir } from "@/lib/manager-paths";
import { randomHex } from "@/lib/random";
import { SESSION_COOKIE_NAME, verifySessionToken } from "@/lib/session";
export const dynamic = "force-dynamic";
/** Hard cap on an uploaded archive. Override with BACKUP_UPLOAD_MAX_BYTES. */
const DEFAULT_MAX_BYTES = 4 * 1024 * 1024 * 1024;
function maxBytes(): number {
const configured = Number(process.env.BACKUP_UPLOAD_MAX_BYTES);
return Number.isFinite(configured) && configured > 0 ? configured : DEFAULT_MAX_BYTES;
}
/**
* Streamed upload of a .7z backup (raw request body, not multipart, so it
* is never buffered). This route is EXCLUDED from proxy.ts's matcher on
* purpose: with a proxy in front, Next.js buffers the request body in
* memory and silently truncates it at proxyClientMaxBodySize (10 MB by
* default), which would corrupt a large archive. So it checks the
* session cookie itself, first thing, before reading a single body byte.
*
* The size cap is enforced twice: up front from Content-Length, and while
* streaming (a client can lie about, or omit, Content-Length).
*/
export async function POST(request: NextRequest) {
const session = verifySessionToken(request.cookies.get(SESSION_COOKIE_NAME)?.value);
if (!session) return NextResponse.json({ error: "Not signed in." }, { status: 401 });
const limit = maxBytes();
const declared = Number(request.headers.get("content-length"));
if (Number.isFinite(declared) && declared > limit) {
return NextResponse.json({ error: `That file is larger than the ${Math.round(limit / 1024 / 1024)} MB upload limit.` }, { status: 413 });
}
if (!request.body) return NextResponse.json({ error: "No file received." }, { status: 400 });
const uploadId = randomHex(8);
const destination = path.join(uploadsDir(), `${uploadId}.7z`);
await mkdir(uploadsDir(), { recursive: true, mode: 0o700 });
let received = 0;
const counter = new Transform({
transform(chunk: Buffer, _encoding, callback) {
received += chunk.length;
if (received > limit) callback(new Error("LIMIT_EXCEEDED"));
else callback(null, chunk);
},
});
try {
await pipeline(Readable.fromWeb(request.body as import("node:stream/web").ReadableStream), counter, createWriteStream(destination, { mode: 0o600 }));
} catch (error) {
await rm(destination, { force: true });
if (error instanceof Error && error.message === "LIMIT_EXCEEDED") {
return NextResponse.json({ error: `That file is larger than the ${Math.round(limit / 1024 / 1024)} MB upload limit.` }, { status: 413 });
}
return NextResponse.json({ error: "The upload was interrupted." }, { status: 400 });
}
if (received === 0) {
await rm(destination, { force: true });
return NextResponse.json({ error: "The uploaded file was empty." }, { status: 400 });
}
return NextResponse.json({ uploadId, sizeBytes: received });
}
+17
View File
@@ -0,0 +1,17 @@
import { NextResponse } from "next/server";
import { readJob } from "@/lib/jobs";
export const dynamic = "force-dynamic";
/**
* Polled by the JobProgress component. Behind the Manager login like
* every route (see proxy.ts). A job record holds step names, a log tail
* and a result summary — never secrets (the restore passphrase only ever
* lives in the running job's closure).
*/
export async function GET(_request: Request, { params }: { params: Promise<{ id: string }> }) {
const { id } = await params;
const job = await readJob(id);
if (!job) return NextResponse.json({ error: "No such job." }, { status: 404 });
return NextResponse.json(job, { headers: { "Cache-Control": "no-store" } });
}
+30 -5
View File
@@ -4,7 +4,12 @@ import path from "node:path";
import { revalidatePath } from "next/cache";
import { composeDown, composeLogs, composeRestart, composeUp } from "@/lib/docker";
import { upsertEnvValues } from "@/lib/env-file";
import { ensureBackupPassphrase, hasBackupPassphrase } from "@/lib/global-settings";
import { setIntentionallyStopped } from "@/lib/health-watcher";
import { fetchInstallationStatus } from "@/lib/installation-status";
import { NO_JOB, type JobStartState } from "@/lib/job-state";
import { assertNotLocked, BusinessLockedError } from "@/lib/locks";
import { startUpdateJob } from "@/lib/operations";
import { randomHex } from "@/lib/random";
import { isValidSlug, stackDir, stackExists } from "@/lib/stacks";
@@ -17,27 +22,47 @@ async function assertStack(slug: string): Promise<string> {
export async function startStackAction(slug: string): Promise<void> {
const dir = await assertStack(slug);
await assertNotLocked(slug);
await composeUp(dir);
await setIntentionallyStopped(slug, false);
revalidatePath(`/dashboard/${slug}`);
}
export async function stopStackAction(slug: string): Promise<void> {
const dir = await assertStack(slug);
await assertNotLocked(slug);
// Mark it first so the health watcher doesn't alert on a stop the operator asked for.
await setIntentionallyStopped(slug, true);
await composeDown(dir);
revalidatePath(`/dashboard/${slug}`);
}
export async function restartStackAction(slug: string): Promise<void> {
const dir = await assertStack(slug);
await assertNotLocked(slug);
await composeRestart(dir);
revalidatePath(`/dashboard/${slug}`);
}
/** Recreates the container against whatever the shared image tag currently points to (--force-recreate) — not a source rebuild. See manager/.env.example's NEXTCHAIR_IMAGE comment for the actual rebuild step. */
export async function updateStackAction(slug: string): Promise<void> {
const dir = await assertStack(slug);
await composeUp(dir, { forceRecreate: true });
revalidatePath(`/dashboard/${slug}`);
/**
* Recreates the container against whatever the shared image tag currently
* points to (--force-recreate) — not a source rebuild. See
* manager/.env.example's NEXTCHAIR_IMAGE comment for the actual rebuild
* step. Runs as a background job (steps: [back up], recreate, wait for
* /api/health, done) the UI follows with <JobProgress>; the per-business
* lock stops it overlapping a backup, restore or delete.
*/
export async function updateStackAction(slug: string, backupFirst: boolean): Promise<JobStartState> {
await assertStack(slug);
try {
// "Back up first" needs a passphrase; if this is the very first backup anywhere, it is minted here and shown once.
const newPassphrase = backupFirst && !(await hasBackupPassphrase()) ? (await ensureBackupPassphrase()).passphrase : null;
const { jobId } = await startUpdateJob(slug, backupFirst);
return { error: null, jobId, newPassphrase };
} catch (error) {
if (error instanceof BusinessLockedError) return { ...NO_JOB, error: error.message };
return { ...NO_JOB, error: error instanceof Error ? error.message : "Couldn't start the update." };
}
}
export async function fetchLogsAction(slug: string): Promise<string> {
@@ -0,0 +1,82 @@
"use server";
import { revalidatePath } from "next/cache";
import { saveBackupSettings, validateBackupSettingsInput, type BackupFrequency } from "@/lib/backup-settings";
import { deleteBackup } from "@/lib/backup-store";
import { ensureBackupPassphrase, hasBackupPassphrase, rotateBackupPassphrase } from "@/lib/global-settings";
import { NO_JOB, type JobStartState } from "@/lib/job-state";
import { assertNotLocked, BusinessLockedError } from "@/lib/locks";
import { startBackupJob } from "@/lib/operations";
import { isValidSlug, stackExists } from "@/lib/stacks";
async function assertBusiness(slug: string): Promise<void> {
if (!isValidSlug(slug) || !(await stackExists(slug))) throw new Error(`No such business: ${slug}`);
}
export interface SaveBackupSettingsState {
error: string | null;
saved: boolean;
/** Plaintext passphrase — only in the one response that just generated it (first enable anywhere). */
newPassphrase: string | null;
}
export async function saveBackupSettingsAction(
slug: string,
_prev: SaveBackupSettingsState,
formData: FormData,
): Promise<SaveBackupSettingsState> {
await assertBusiness(slug);
const input = {
enabled: formData.get("enabled") === "on",
frequency: (formData.get("frequency") === "weekly" ? "weekly" : "daily") as BackupFrequency,
time: String(formData.get("time") ?? "").trim(),
weekday: Number(formData.get("weekday") ?? 0),
keep: Number(formData.get("keep") ?? 2),
};
const problem = validateBackupSettingsInput(input);
if (problem) return { error: problem, saved: false, newPassphrase: null };
// Generated at the first enable, encrypted with the Manager's key, shown once.
let newPassphrase: string | null = null;
if (input.enabled) {
const ensured = await ensureBackupPassphrase();
if (ensured.created) newPassphrase = ensured.passphrase;
}
await saveBackupSettings(slug, input);
revalidatePath(`/dashboard/${slug}/backups`);
return { error: null, saved: true, newPassphrase };
}
export async function backupNowAction(slug: string): Promise<JobStartState> {
await assertBusiness(slug);
try {
const newPassphrase = (await hasBackupPassphrase()) ? null : (await ensureBackupPassphrase()).passphrase;
const { jobId } = await startBackupJob(slug, "regular");
return { error: null, jobId, newPassphrase };
} catch (error) {
if (error instanceof BusinessLockedError) return { ...NO_JOB, error: error.message };
return { ...NO_JOB, error: error instanceof Error ? error.message : "Couldn't start the backup." };
}
}
export async function deleteBackupAction(slug: string, fileName: string): Promise<{ error: string | null }> {
try {
await assertNotLocked(slug); // a running backup/restore might be reading or pruning files
await deleteBackup(slug, fileName);
revalidatePath(`/dashboard/${slug}/backups`);
return { error: null };
} catch (error) {
return { error: error instanceof Error ? error.message : "Couldn't delete that backup." };
}
}
/** Replaces the passphrase used for NEW backups. Older archives still need the old one — the UI says so. Returns the new value once. */
export async function rotatePassphraseAction(): Promise<{ error: string | null; newPassphrase: string | null }> {
try {
return { error: null, newPassphrase: await rotateBackupPassphrase() };
} catch (error) {
return { error: error instanceof Error ? error.message : "Couldn't generate a passphrase.", newPassphrase: null };
}
}
@@ -0,0 +1,268 @@
"use client";
import Link from "next/link";
import { useRouter } from "next/navigation";
import { useActionState, useState, useTransition } from "react";
import type { BackupEntry } from "@/lib/backup-store";
import { JobProgress } from "../../job-progress";
import { PassphraseNotice } from "../../passphrase-notice";
import {
backupNowAction,
deleteBackupAction,
rotatePassphraseAction,
saveBackupSettingsAction,
type SaveBackupSettingsState,
} from "./actions";
interface SettingsView {
enabled: boolean;
frequency: "daily" | "weekly";
time: string;
weekday: number;
keep: number;
lastAttemptAt: string | null;
lastSuccessAt: string | null;
lastStatus: "ok" | "failed" | null;
lastMessage: string | null;
}
const WEEKDAYS = ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"];
const initialSaveState: SaveBackupSettingsState = { error: null, saved: false, newPassphrase: null };
function formatBytes(bytes: number): string {
if (bytes < 1024 * 1024) return `${Math.max(1, Math.round(bytes / 1024))} KB`;
if (bytes < 1024 * 1024 * 1024) return `${(bytes / 1024 / 1024).toFixed(1)} MB`;
return `${(bytes / 1024 / 1024 / 1024).toFixed(2)} GB`;
}
function formatWhen(iso: string | null): string {
if (!iso) return "never";
return new Date(iso).toLocaleString();
}
const KIND_LABEL: Record<BackupEntry["kind"], string> = {
regular: "Backup",
safety: "Safety (auto, before update/restore)",
final: "Final (before delete)",
};
export function BackupsPanel({
slug,
settings,
backups,
hasPassphrase,
passphraseCreatedAt,
activeJobId,
timeZone,
}: {
slug: string;
settings: SettingsView;
backups: BackupEntry[];
hasPassphrase: boolean;
passphraseCreatedAt: string | null;
activeJobId: string | null;
timeZone: string;
}) {
const router = useRouter();
const [saveState, saveAction, isSaving] = useActionState(saveBackupSettingsAction.bind(null, slug), initialSaveState);
const [frequency, setFrequency] = useState(settings.frequency);
const [isPending, startTransition] = useTransition();
const [error, setError] = useState<string | null>(null);
const [jobId, setJobId] = useState<string | null>(activeJobId);
const [jobRunning, setJobRunning] = useState(activeJobId !== null);
const [shownPassphrase, setShownPassphrase] = useState<string | null>(null);
const passphraseToShow = shownPassphrase ?? saveState.newPassphrase;
const backupNow = () => {
setError(null);
startTransition(async () => {
const result = await backupNowAction(slug);
if (result.error) {
setError(result.error);
return;
}
if (result.newPassphrase) setShownPassphrase(result.newPassphrase);
setJobId(result.jobId);
setJobRunning(true);
});
};
const remove = (fileName: string) => {
if (!window.confirm(`Delete ${fileName}? This cannot be undone.`)) return;
setError(null);
startTransition(async () => {
const result = await deleteBackupAction(slug, fileName);
if (result.error) setError(result.error);
else router.refresh();
});
};
const rotate = () => {
if (
!window.confirm(
"Generate a NEW passphrase? New backups will use it. Existing backups keep the old one — you will need both to open all of them.",
)
) {
return;
}
startTransition(async () => {
const result = await rotatePassphraseAction();
if (result.error) setError(result.error);
else setShownPassphrase(result.newPassphrase);
});
};
return (
<div className="space-y-8">
{passphraseToShow && <PassphraseNotice passphrase={passphraseToShow} />}
<section className="space-y-3">
<h2 className="text-lg font-semibold">Schedule</h2>
<form action={saveAction} className="space-y-4 rounded-md border border-line bg-surface p-5">
<label className="flex items-center gap-2 text-sm font-medium">
<input type="checkbox" name="enabled" defaultChecked={settings.enabled} />
Back up this business automatically
</label>
<div className="grid grid-cols-1 gap-4 sm:grid-cols-4">
<div>
<label htmlFor="frequency" className="block text-sm font-medium text-muted">
How often
</label>
<select
id="frequency"
name="frequency"
value={frequency}
onChange={(e) => setFrequency(e.target.value as "daily" | "weekly")}
className="mt-1 w-full rounded-md border border-line px-3 py-2 text-sm"
>
<option value="daily">Daily</option>
<option value="weekly">Weekly</option>
</select>
</div>
{frequency === "weekly" && (
<div>
<label htmlFor="weekday" className="block text-sm font-medium text-muted">
Day
</label>
<select id="weekday" name="weekday" defaultValue={settings.weekday} className="mt-1 w-full rounded-md border border-line px-3 py-2 text-sm">
{WEEKDAYS.map((day, index) => (
<option key={day} value={index}>
{day}
</option>
))}
</select>
</div>
)}
<div>
<label htmlFor="time" className="block text-sm font-medium text-muted">
Time ({timeZone})
</label>
<input id="time" name="time" type="time" required defaultValue={settings.time} className="mt-1 w-full rounded-md border border-line px-3 py-2 text-sm" />
</div>
<div>
<label htmlFor="keep" className="block text-sm font-medium text-muted">
Backups to keep
</label>
<input
id="keep"
name="keep"
type="number"
min={1}
max={14}
required
defaultValue={settings.keep}
className="mt-1 w-full rounded-md border border-line px-3 py-2 text-sm"
/>
</div>
</div>
{saveState.error && (
<p role="alert" className="text-sm font-medium text-danger">
{saveState.error}
</p>
)}
{saveState.saved && !saveState.error && <p className="text-sm text-green-700">Saved.</p>}
<button type="submit" disabled={isSaving} className="rounded-md bg-ink px-4 py-2 text-sm font-semibold text-white disabled:opacity-60">
{isSaving ? "Saving…" : "Save schedule"}
</button>
<p className="text-xs text-muted">
Last backup run: {formatWhen(settings.lastAttemptAt)}
{settings.lastStatus === "ok" && " — succeeded"}
{settings.lastStatus === "failed" && <span className="text-danger"> — failed: {settings.lastMessage}</span>}
</p>
</form>
</section>
<section className="space-y-3">
<div className="flex items-center justify-between">
<h2 className="text-lg font-semibold">Backups</h2>
<button
type="button"
onClick={backupNow}
disabled={isPending || jobRunning}
className="rounded-md bg-ink px-4 py-2 text-sm font-semibold text-white disabled:opacity-60"
>
Back up now
</button>
</div>
{error && (
<p role="alert" className="text-sm font-medium text-danger">
{error}
</p>
)}
{jobId && <JobProgress key={jobId} jobId={jobId} onFinished={() => setJobRunning(false)} />}
{backups.length === 0 ? (
<p className="rounded-md border border-line bg-surface p-4 text-sm text-muted">No backups yet.</p>
) : (
<ul className="divide-y divide-line rounded-md border border-line bg-surface">
{backups.map((backup) => (
<li key={backup.fileName} className="flex flex-wrap items-center gap-3 px-4 py-3 text-sm">
<div className="min-w-0 flex-1">
<p className="truncate font-mono text-xs">{backup.fileName}</p>
<p className="text-xs text-muted">
{new Date(backup.createdAt).toLocaleString()} · {formatBytes(backup.sizeBytes)} · {KIND_LABEL[backup.kind]}
</p>
</div>
<a
href={`/api/backups/${slug}/${backup.fileName}`}
className="rounded-md border border-line px-3 py-1 text-xs font-medium"
download
>
Download
</a>
<Link
href={`/dashboard/restore?slug=${encodeURIComponent(slug)}&file=${encodeURIComponent(backup.fileName)}`}
className="rounded-md border border-line px-3 py-1 text-xs font-medium"
>
Restore…
</Link>
<button
type="button"
disabled={isPending || jobRunning}
onClick={() => remove(backup.fileName)}
className="rounded-md border border-danger/30 px-3 py-1 text-xs font-medium text-danger disabled:opacity-60"
>
Delete
</button>
</li>
))}
</ul>
)}
</section>
<section className="space-y-2 text-sm text-muted">
<h2 className="text-lg font-semibold text-ink">Passphrase</h2>
<p>
{hasPassphrase
? `A passphrase protects every backup${passphraseCreatedAt ? ` (created ${new Date(passphraseCreatedAt).toLocaleDateString()})` : ""}. It is stored encrypted here and cannot be shown again — keep your saved copy; a fresh server needs it to restore.`
: "No passphrase yet — one is generated (and shown once) the first time you enable backups or press Back up now."}
</p>
{hasPassphrase && (
<button type="button" onClick={rotate} disabled={isPending} className="font-medium text-ink underline disabled:opacity-60">
Generate a new passphrase…
</button>
)}
</section>
</div>
);
}
@@ -0,0 +1,66 @@
import type { Metadata } from "next";
import Link from "next/link";
import { notFound } from "next/navigation";
import { readBackupSettings } from "@/lib/backup-settings";
import { listBackups } from "@/lib/backup-store";
import { backupPassphraseCreatedAt, hasBackupPassphrase } from "@/lib/global-settings";
import { getActiveLock } from "@/lib/locks";
import { listBusinessStacks } from "@/lib/stacks";
import { BackupsPanel } from "./backups-panel";
export const metadata: Metadata = { title: "Backups" };
export const dynamic = "force-dynamic";
export default async function BackupsPage({ params }: { params: Promise<{ slug: string }> }) {
const { slug } = await params;
const stacks = await listBusinessStacks();
const stack = stacks.find((s) => s.slug === slug);
if (!stack) notFound();
const [settings, backups, hasPassphrase, passphraseCreatedAt, lock] = await Promise.all([
readBackupSettings(slug),
listBackups(slug),
hasBackupPassphrase(),
backupPassphraseCreatedAt(),
getActiveLock(slug),
]);
return (
<div className="space-y-6">
<div>
<Link href={`/dashboard/${slug}`} className="text-sm text-muted hover:text-ink">
← {stack.name ?? slug}
</Link>
<h1 className="mt-2 text-xl font-bold">Backups</h1>
<p className="text-sm text-muted">
Encrypted 7z archives of the database, media files and configuration, stored on this server under{" "}
<span className="font-mono">backups/{slug}/</span>. To restore one — onto this business or a fresh server — use{" "}
<Link href="/dashboard/restore" className="font-medium text-ink underline">
Restore
</Link>
.
</p>
</div>
<BackupsPanel
slug={slug}
settings={{
enabled: settings.enabled,
frequency: settings.frequency,
time: settings.time,
weekday: settings.weekday,
keep: settings.keep,
lastAttemptAt: settings.lastAttemptAt,
lastSuccessAt: settings.lastSuccessAt,
lastStatus: settings.lastStatus,
lastMessage: settings.lastMessage,
}}
backups={backups}
hasPassphrase={hasPassphrase}
passphraseCreatedAt={passphraseCreatedAt}
activeJobId={lock?.jobId ?? null}
timeZone={Intl.DateTimeFormat().resolvedOptions().timeZone}
/>
</div>
);
}
@@ -1,6 +1,8 @@
"use client";
import { useState, useTransition } from "react";
import { JobProgress } from "../job-progress";
import { PassphraseNotice } from "../passphrase-notice";
import {
fetchLogsAction,
restartStackAction,
@@ -9,10 +11,18 @@ import {
updateStackAction,
} from "./actions";
export function BusinessActions({ slug }: { slug: string }) {
export function BusinessActions({ slug, activeJobId }: { slug: string; activeJobId: string | null }) {
const [isPending, startTransition] = useTransition();
const [error, setError] = useState<string | null>(null);
const [logs, setLogs] = useState<string | null>(null);
const [updateOpen, setUpdateOpen] = useState(false);
const [backupFirst, setBackupFirst] = useState(true);
const [jobId, setJobId] = useState<string | null>(activeJobId);
const [jobRunning, setJobRunning] = useState(activeJobId !== null);
const [newPassphrase, setNewPassphrase] = useState<string | null>(null);
// Quick actions are refused server-side while a job holds the business; the buttons just mirror that.
const busy = isPending || jobRunning;
const run = (action: (slug: string) => Promise<void>) => {
setError(null);
@@ -25,6 +35,25 @@ export function BusinessActions({ slug }: { slug: string }) {
});
};
const runUpdate = () => {
setError(null);
startTransition(async () => {
try {
const result = await updateStackAction(slug, backupFirst);
if (result.error) {
setError(result.error);
return;
}
setNewPassphrase(result.newPassphrase);
setJobId(result.jobId);
setJobRunning(true);
setUpdateOpen(false);
} catch (e) {
setError(e instanceof Error ? e.message : "Couldn't start the update.");
}
});
};
const loadLogs = () => {
setError(null);
startTransition(async () => {
@@ -41,7 +70,7 @@ export function BusinessActions({ slug }: { slug: string }) {
<div className="flex flex-wrap gap-2">
<button
type="button"
disabled={isPending}
disabled={busy}
onClick={() => run(startStackAction)}
className="rounded-md border border-line px-3 py-1.5 text-sm font-medium disabled:opacity-60"
>
@@ -49,7 +78,7 @@ export function BusinessActions({ slug }: { slug: string }) {
</button>
<button
type="button"
disabled={isPending}
disabled={busy}
onClick={() => run(stopStackAction)}
className="rounded-md border border-line px-3 py-1.5 text-sm font-medium disabled:opacity-60"
>
@@ -57,7 +86,7 @@ export function BusinessActions({ slug }: { slug: string }) {
</button>
<button
type="button"
disabled={isPending}
disabled={busy}
onClick={() => run(restartStackAction)}
className="rounded-md border border-line px-3 py-1.5 text-sm font-medium disabled:opacity-60"
>
@@ -65,8 +94,9 @@ export function BusinessActions({ slug }: { slug: string }) {
</button>
<button
type="button"
disabled={isPending}
onClick={() => run(updateStackAction)}
disabled={busy}
onClick={() => setUpdateOpen((open) => !open)}
aria-expanded={updateOpen}
className="rounded-md border border-line px-3 py-1.5 text-sm font-medium disabled:opacity-60"
title="Recreates the container against the current shared image tag — not a source rebuild."
>
@@ -81,11 +111,36 @@ export function BusinessActions({ slug }: { slug: string }) {
View logs
</button>
</div>
{updateOpen && !jobRunning && (
<div className="space-y-3 rounded-md border border-line bg-surface p-4">
<p className="text-sm text-muted">
Recreates the container against the current shared image tag, then waits for it to answer{" "}
<span className="font-mono">/api/health</span>. This does <strong>not</strong> rebuild the image — build{" "}
<span className="font-mono">nextchair-app</span> first if the code changed.
</p>
<label className="flex items-center gap-2 text-sm">
<input type="checkbox" checked={backupFirst} onChange={(e) => setBackupFirst(e.target.checked)} />
Back up first (recommended)
</label>
<button
type="button"
disabled={isPending}
onClick={runUpdate}
className="rounded-md bg-ink px-4 py-2 text-sm font-semibold text-white disabled:opacity-60"
>
{isPending ? "Starting…" : "Run update"}
</button>
</div>
)}
{error && (
<p role="alert" className="text-sm font-medium text-danger">
{error}
</p>
)}
{newPassphrase && <PassphraseNotice passphrase={newPassphrase} />}
{jobId && <JobProgress key={jobId} jobId={jobId} onFinished={() => setJobRunning(false)} />}
{logs !== null && (
<pre className="max-h-96 overflow-auto rounded-md bg-ink p-4 text-xs text-white">{logs}</pre>
)}
@@ -1,16 +1,25 @@
"use server";
import { redirect } from "next/navigation";
import { deleteBusiness } from "@/lib/provisioning";
import { ensureBackupPassphrase, hasBackupPassphrase } from "@/lib/global-settings";
import { BusinessLockedError } from "@/lib/locks";
import { startDeleteJob } from "@/lib/operations";
export interface DeleteBusinessState {
error: string | null;
/** Set once the delete job has started — the form then follows it with <JobProgress>. */
jobId: string | null;
/** Plaintext backup passphrase, only in the response that just generated it. */
newPassphrase: string | null;
}
/**
* `slug` is bound in by the page, the same way updateBusinessAction takes
* it — the operator only ever types a *confirmation* string, never the
* slug being acted on as a submittable field.
*
* Deleting is a background job holding the business lock (so it can't
* overlap an update, backup or restore). With "final backup" ticked the
* job takes one first and aborts WITHOUT deleting if that backup fails.
*/
export async function deleteBusinessAction(
slug: string,
@@ -19,17 +28,16 @@ export async function deleteBusinessAction(
): Promise<DeleteBusinessState> {
const confirmation = String(formData.get("confirmation") ?? "");
if (confirmation !== slug) {
return { error: `Type "${slug}" exactly to confirm deletion.` };
return { error: `Type "${slug}" exactly to confirm deletion.`, jobId: null, newPassphrase: null };
}
const finalBackup = formData.get("finalBackup") === "on";
let result: { slug: string; warnings: string[] };
try {
result = await deleteBusiness(slug);
const newPassphrase = finalBackup && !(await hasBackupPassphrase()) ? (await ensureBackupPassphrase()).passphrase : null;
const { jobId } = await startDeleteJob(slug, finalBackup);
return { error: null, jobId, newPassphrase };
} catch (error) {
return { error: error instanceof Error ? error.message : "Failed to delete this business." };
if (error instanceof BusinessLockedError) return { error: error.message, jobId: null, newPassphrase: null };
return { error: error instanceof Error ? error.message : "Failed to delete this business.", jobId: null, newPassphrase: null };
}
const params = new URLSearchParams({ justDeleted: result.slug });
if (result.warnings.length > 0) params.set("warnings", result.warnings.join("; "));
redirect(`/dashboard?${params.toString()}`);
}
@@ -1,9 +1,12 @@
"use client";
import Link from "next/link";
import { useActionState, useState } from "react";
import { JobProgress } from "../../job-progress";
import { PassphraseNotice } from "../../passphrase-notice";
import { deleteBusinessAction, type DeleteBusinessState } from "./actions";
const initialState: DeleteBusinessState = { error: null };
const initialState: DeleteBusinessState = { error: null, jobId: null, newPassphrase: null };
export function DeleteBusinessForm({ slug }: { slug: string }) {
const boundAction = deleteBusinessAction.bind(null, slug);
@@ -11,8 +14,27 @@ export function DeleteBusinessForm({ slug }: { slug: string }) {
const [confirmation, setConfirmation] = useState("");
const canDelete = confirmation === slug;
if (state.jobId) {
return (
<div className="space-y-4">
{state.newPassphrase && <PassphraseNotice passphrase={state.newPassphrase} />}
<JobProgress jobId={state.jobId} refreshOnFinish={false}>
<Link href="/dashboard" className="inline-block rounded-md bg-ink px-4 py-2 text-sm font-semibold text-white">
Back to businesses
</Link>
</JobProgress>
</div>
);
}
return (
<form action={formAction} className="space-y-4">
<label className="flex items-start gap-2 text-sm">
<input type="checkbox" name="finalBackup" defaultChecked className="mt-0.5" />
<span>
Take a final backup first <span className="text-muted">(recommended — kept on this server after the business is gone; deleting stops if the backup fails)</span>
</span>
</label>
<div>
<label htmlFor="confirmation" className="block text-sm font-medium text-muted">
Type <span className="font-mono font-semibold text-ink">{slug}</span> to confirm
@@ -39,7 +61,7 @@ export function DeleteBusinessForm({ slug }: { slug: string }) {
disabled={!canDelete || isPending}
className="w-full rounded-md bg-danger py-2 text-sm font-semibold text-white disabled:opacity-40"
>
{isPending ? "Deleting…" : "Delete Business"}
{isPending ? "Starting…" : "Delete Business"}
</button>
</form>
);
+10 -1
View File
@@ -1,6 +1,7 @@
import Link from "next/link";
import { notFound } from "next/navigation";
import { resolveInstallationView } from "@/lib/installation-status";
import { getActiveLock } from "@/lib/locks";
import { listBusinessStacks } from "@/lib/stacks";
import { BusinessActions } from "./business-actions";
import { InstallationTokenPanel } from "./installation-token-panel";
@@ -22,6 +23,8 @@ export default async function BusinessDetailPage({
if (!stack) notFound();
const installationView = await resolveInstallationView(slug, stack.domain);
// A job already running for this business (started in another tab, or before a page reload) — followed live below.
const activeLock = await getActiveLock(slug);
return (
<div className="space-y-6">
@@ -46,6 +49,12 @@ export default async function BusinessDetailPage({
<p className="text-sm text-muted">{stack.domain ?? "no domain set"}</p>
</div>
<div className="flex gap-2">
<Link
href={`/dashboard/${stack.slug}/backups`}
className="rounded-md border border-line px-3 py-1.5 text-sm font-medium"
>
Backups
</Link>
<Link
href={`/dashboard/${stack.slug}/edit`}
className="rounded-md border border-line px-3 py-1.5 text-sm font-medium"
@@ -61,7 +70,7 @@ export default async function BusinessDetailPage({
</div>
</div>
<BusinessActions slug={stack.slug} />
<BusinessActions slug={stack.slug} activeJobId={activeLock?.jobId ?? null} />
<InstallationTokenPanel slug={stack.slug} view={installationView} />
</div>
+153
View File
@@ -0,0 +1,153 @@
"use client";
import { useRouter } from "next/navigation";
import { useEffect, useRef, useState } from "react";
import type { JobRecord, StepStatus } from "@/lib/jobs";
const POLL_MS = 1500;
const STEP_GLYPH: Record<StepStatus, string> = {
pending: "○",
running: "◐",
done: "✓",
failed: "✕",
skipped: "–",
};
const STEP_COLOR: Record<StepStatus, string> = {
pending: "text-muted",
running: "text-accent",
done: "text-accent",
failed: "text-danger",
skipped: "text-muted",
};
/**
* Live progress for a background job: polls GET /api/jobs/<id> every
* 1.5s until the job finishes, showing each step's state and the log
* tail. Used by Update, Backup, Restore and Delete.
*/
export function JobProgress({
jobId,
onFinished,
refreshOnFinish = true,
children,
}: {
jobId: string;
/** Re-render the surrounding server page when the job ends. Off for Delete, whose page 404s once the business is gone. */
refreshOnFinish?: boolean;
/** Called once, when the job reaches succeeded/failed. */
onFinished?: (job: JobRecord) => void;
/** Rendered under the steps once the job has finished successfully (e.g. a link onward). */
children?: React.ReactNode;
}) {
const router = useRouter();
const [job, setJob] = useState<JobRecord | null>(null);
const [pollError, setPollError] = useState<string | null>(null);
const finishedRef = useRef(false);
const onFinishedRef = useRef(onFinished);
useEffect(() => {
onFinishedRef.current = onFinished;
}, [onFinished]);
useEffect(() => {
finishedRef.current = false;
let cancelled = false;
let timer: ReturnType<typeof setTimeout> | undefined;
async function poll() {
try {
const response = await fetch(`/api/jobs/${encodeURIComponent(jobId)}`, { cache: "no-store" });
if (response.status === 404) {
if (!cancelled) setPollError("This job's record is gone (it may have been cleaned up).");
return;
}
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const next = (await response.json()) as JobRecord;
if (cancelled) return;
setJob(next);
setPollError(null);
if (next.status !== "running") {
if (!finishedRef.current) {
finishedRef.current = true;
onFinishedRef.current?.(next);
if (refreshOnFinish) router.refresh();
}
return;
}
} catch (error) {
// A restart of the Manager itself drops polls for a moment — keep trying.
if (!cancelled) setPollError(error instanceof Error ? error.message : "Couldn't reach the Manager.");
}
if (!cancelled) timer = setTimeout(poll, POLL_MS);
}
void poll();
return () => {
cancelled = true;
if (timer) clearTimeout(timer);
};
}, [jobId, router, refreshOnFinish]);
if (!job) {
return <p className="text-sm text-muted">{pollError ?? "Starting…"}</p>;
}
return (
<div className="space-y-3 rounded-md border border-line bg-surface p-4" aria-live="polite">
<div className="flex items-center justify-between">
<h3 className="text-sm font-semibold">{job.title}</h3>
<span
className={`rounded-full px-3 py-1 text-xs font-semibold ${
job.status === "running"
? "bg-stone text-muted"
: job.status === "succeeded"
? "bg-green-100 text-green-800"
: "bg-danger/10 text-danger"
}`}
>
{job.status === "running" ? "Running…" : job.status === "succeeded" ? "Done" : "Failed"}
</span>
</div>
<ol className="space-y-1.5">
{job.steps.map((step) => (
<li key={step.id} className="flex items-start gap-2 text-sm">
<span className={`w-4 shrink-0 text-center font-mono ${STEP_COLOR[step.status]}`} aria-hidden="true">
{STEP_GLYPH[step.status]}
</span>
<span className={step.status === "pending" || step.status === "skipped" ? "text-muted" : undefined}>
{step.label}
<span className="sr-only"> — {step.status}</span>
{step.detail && <span className="ml-2 text-xs text-muted">{step.detail}</span>}
</span>
</li>
))}
</ol>
{job.error && (
<p role="alert" className="text-sm font-medium text-danger">
{job.error}
</p>
)}
{job.status === "succeeded" && job.result && (
<p className="text-xs text-muted">
{Object.entries(job.result)
.filter(([, value]) => value !== null && value !== "")
.map(([key, value]) => `${key}: ${String(value)}`)
.join(" · ")}
</p>
)}
{pollError && job.status === "running" && <p className="text-xs text-muted">Reconnecting… ({pollError})</p>}
{job.logTail.length > 0 && (
<details>
<summary className="cursor-pointer text-xs font-medium text-muted">Log</summary>
<pre className="mt-2 max-h-56 overflow-auto rounded-md bg-ink p-3 text-xs text-white">{job.logTail.join("\n")}</pre>
</details>
)}
{job.status === "succeeded" && children}
</div>
);
}
+3
View File
@@ -9,6 +9,9 @@ export default function DashboardLayout({ children }: { children: React.ReactNod
NextChair Manager
</Link>
<div className="flex items-center gap-4 text-sm">
<Link href="/dashboard/restore" className="text-muted hover:text-ink">
Restore
</Link>
<Link href="/dashboard/settings" className="text-muted hover:text-ink">
Settings
</Link>
@@ -0,0 +1,33 @@
"use client";
import { useState } from "react";
/** The backup passphrase, shown exactly once (in the response that generated it). Nothing here persists it — the server keeps only an encrypted copy. */
export function PassphraseNotice({ passphrase }: { passphrase: string }) {
const [copied, setCopied] = useState(false);
async function copy() {
try {
await navigator.clipboard.writeText(passphrase);
setCopied(true);
} catch {
setCopied(false);
}
}
return (
<div role="alert" className="space-y-2 rounded-md border border-amber-300 bg-amber-50 p-4 text-sm">
<p className="font-semibold text-amber-900">Save this backup passphrase now — it won&apos;t be shown again.</p>
<p className="text-amber-900">
Every backup is encrypted with it. On a new server you&apos;ll need to type it to restore; without it the backups
cannot be opened by anyone.
</p>
<div className="flex flex-wrap items-center gap-3">
<code className="select-all rounded bg-white px-3 py-1.5 font-mono text-base font-semibold">{passphrase}</code>
<button type="button" onClick={copy} className="rounded-md border border-line bg-white px-3 py-1.5 text-sm font-medium">
{copied ? "Copied" : "Copy"}
</button>
</div>
</div>
);
}
@@ -0,0 +1,104 @@
"use server";
import { ensureBackupPassphrase, hasBackupPassphrase, readBackupPassphrase } from "@/lib/global-settings";
import { BusinessLockedError } from "@/lib/locks";
import { startRestoreJob } from "@/lib/operations";
import { planRestore, readArchiveManifest, resolveArchivePath, type RestoreSource } from "@/lib/restore";
/** "backup:<slug>:<fileName>" for a stored archive, "upload:<id>" for one just uploaded. Parsed strictly; resolveArchivePath re-validates both forms. */
function parseSourceKey(key: string): RestoreSource {
if (key.startsWith("upload:")) return { kind: "upload", uploadId: key.slice("upload:".length) };
const match = /^backup:([^:]+):([^:]+)$/.exec(key);
if (!match) throw new Error("Choose a backup to restore.");
return { kind: "backup", slug: match[1]!, fileName: match[2]! };
}
export interface RestorePreview {
slug: string;
name: string | null;
domain: string;
createdAt: string;
kind: string;
lastMigration: string | null;
slugExists: boolean;
domainConflict: string | null;
schemaProblem: string | null;
}
export interface InspectRestoreState {
error: string | null;
preview: RestorePreview | null;
}
async function passphraseFor(input: string): Promise<string> {
const passphrase = input.trim() || (await readBackupPassphrase()) || "";
if (!passphrase) throw new Error("Enter the backup passphrase (this server has no stored one).");
return passphrase;
}
/** Opens only the encrypted header (fast) and reports what restoring would do — nothing is changed. A wrong passphrase fails here. */
export async function inspectRestoreAction(sourceKey: string, passphraseInput: string): Promise<InspectRestoreState> {
try {
const source = parseSourceKey(sourceKey);
const manifest = await readArchiveManifest(resolveArchivePath(source), await passphraseFor(passphraseInput));
const plan = await planRestore(manifest);
return {
error: null,
preview: {
slug: manifest.slug,
name: manifest.name,
domain: manifest.domain,
createdAt: manifest.createdAt,
kind: manifest.kind,
lastMigration: manifest.lastMigration,
slugExists: plan.slugExists,
domainConflict: plan.domainConflict,
schemaProblem: plan.schema.ok ? null : plan.schema.reason,
},
};
} catch (error) {
// A wrong passphrase makes 7zz fail with a generic message; say so plainly instead of passing its output through.
const message = error instanceof Error ? error.message : "";
const looksLikePassphrase = /wrong password|can not open encrypted|data error|headers error|exited with code 2/i.test(message);
return {
error: looksLikePassphrase ? "Couldn't open the archive — wrong passphrase, or the file isn't a valid NextChair backup." : message || "Couldn't read that backup.",
preview: null,
};
}
}
export interface StartRestoreState {
error: string | null;
jobId: string | null;
newPassphrase: string | null;
}
export async function startRestoreAction(
sourceKey: string,
passphraseInput: string,
replace: boolean,
confirmSlug: string,
): Promise<StartRestoreState> {
const fail = (error: string): StartRestoreState => ({ error, jobId: null, newPassphrase: null });
try {
const source = parseSourceKey(sourceKey);
const passphrase = await passphraseFor(passphraseInput);
const manifest = await readArchiveManifest(resolveArchivePath(source), passphrase);
// Re-checked here, not trusted from the preview the browser holds.
const plan = await planRestore(manifest);
if (plan.domainConflict) return fail(plan.domainConflict);
if (!plan.schema.ok) return fail(plan.schema.reason);
if (plan.slugExists && (!replace || confirmSlug !== manifest.slug)) {
return fail(`"${manifest.slug}" already exists. Tick "replace" and type its slug to confirm overwriting it.`);
}
// Replacing takes a safety backup, which needs a stored passphrase; mint one (shown once) if this server has none yet.
const newPassphrase = plan.slugExists && !(await hasBackupPassphrase()) ? (await ensureBackupPassphrase()).passphrase : null;
const { jobId } = await startRestoreJob({ source, passphrase, replace });
return { error: null, jobId, newPassphrase };
} catch (error) {
if (error instanceof BusinessLockedError) return fail(error.message);
return fail(error instanceof Error ? error.message : "Couldn't start the restore.");
}
}
@@ -0,0 +1,45 @@
import type { Metadata } from "next";
import Link from "next/link";
import { listAllBackups } from "@/lib/backup-store";
import { hasBackupPassphrase } from "@/lib/global-settings";
import { RestoreWizard } from "./restore-wizard";
export const metadata: Metadata = { title: "Restore" };
export const dynamic = "force-dynamic";
export default async function RestorePage({ searchParams }: { searchParams: Promise<{ slug?: string; file?: string }> }) {
const { slug, file } = await searchParams;
const [backups, hasPassphrase] = await Promise.all([listAllBackups(), hasBackupPassphrase()]);
const preselected = slug && file ? `backup:${slug}:${file}` : null;
const configuredMax = Number(process.env.BACKUP_UPLOAD_MAX_BYTES);
const maxUploadBytes = configuredMax > 0 ? configuredMax : 4 * 1024 * 1024 * 1024;
return (
<div className="max-w-2xl space-y-6">
<div>
<Link href="/dashboard" className="text-sm text-muted hover:text-ink">
← Businesses
</Link>
<h1 className="mt-2 text-xl font-bold">Restore a business</h1>
<p className="text-sm text-muted">
Restores a backup onto the business it came from, or onto a fresh server where it never existed. Pick a stored
backup or upload a <span className="font-mono">.7z</span>, enter the passphrase, check what it contains, then
confirm.
</p>
</div>
<RestoreWizard
backups={backups.map((b) => ({
key: `backup:${b.slug}:${b.fileName}`,
slug: b.slug,
fileName: b.fileName,
createdAt: b.createdAt,
kind: b.kind,
sizeBytes: b.sizeBytes,
}))}
preselected={preselected}
hasStoredPassphrase={hasPassphrase}
maxUploadBytes={maxUploadBytes}
/>
</div>
);
}
@@ -0,0 +1,296 @@
"use client";
import Link from "next/link";
import { useRef, useState, useTransition } from "react";
import { JobProgress } from "../job-progress";
import { PassphraseNotice } from "../passphrase-notice";
import { inspectRestoreAction, startRestoreAction, type RestorePreview } from "./actions";
interface BackupOption {
key: string;
slug: string;
fileName: string;
createdAt: string;
kind: string;
sizeBytes: number;
}
function formatBytes(bytes: number): string {
if (bytes < 1024 * 1024) return `${Math.max(1, Math.round(bytes / 1024))} KB`;
if (bytes < 1024 * 1024 * 1024) return `${(bytes / 1024 / 1024).toFixed(1)} MB`;
return `${(bytes / 1024 / 1024 / 1024).toFixed(2)} GB`;
}
export function RestoreWizard({
backups,
preselected,
hasStoredPassphrase,
maxUploadBytes,
}: {
backups: BackupOption[];
preselected: string | null;
hasStoredPassphrase: boolean;
maxUploadBytes: number;
}) {
const [mode, setMode] = useState<"stored" | "upload">(backups.length === 0 ? "upload" : "stored");
const [selected, setSelected] = useState(preselected ?? backups[0]?.key ?? "");
const [uploadKey, setUploadKey] = useState<string | null>(null);
const [uploadProgress, setUploadProgress] = useState<number | null>(null);
const [uploadName, setUploadName] = useState<string | null>(null);
const [passphrase, setPassphrase] = useState("");
const [preview, setPreview] = useState<RestorePreview | null>(null);
const [error, setError] = useState<string | null>(null);
const [replace, setReplace] = useState(false);
const [confirmSlug, setConfirmSlug] = useState("");
const [jobId, setJobId] = useState<string | null>(null);
const [restoredSlug, setRestoredSlug] = useState<string | null>(null);
const [newPassphrase, setNewPassphrase] = useState<string | null>(null);
const [isPending, startTransition] = useTransition();
const xhrRef = useRef<XMLHttpRequest | null>(null);
const sourceKey = mode === "upload" ? uploadKey : selected;
const resetPreview = () => {
setPreview(null);
setError(null);
};
function upload(file: File) {
resetPreview();
setUploadKey(null);
if (file.size > maxUploadBytes) {
setError(`That file is ${formatBytes(file.size)}, over the ${formatBytes(maxUploadBytes)} upload limit. Copy it into the server's backups folder instead.`);
return;
}
setUploadName(file.name);
setUploadProgress(0);
const xhr = new XMLHttpRequest();
xhrRef.current = xhr;
xhr.open("POST", "/api/backups/upload");
xhr.setRequestHeader("Content-Type", "application/octet-stream");
xhr.upload.onprogress = (event) => {
if (event.lengthComputable) setUploadProgress(Math.round((event.loaded / event.total) * 100));
};
xhr.onload = () => {
setUploadProgress(null);
try {
const body = JSON.parse(xhr.responseText) as { uploadId?: string; error?: string };
if (xhr.status === 200 && body.uploadId) setUploadKey(`upload:${body.uploadId}`);
else setError(body.error ?? "Upload failed.");
} catch {
setError("Upload failed.");
}
};
xhr.onerror = () => {
setUploadProgress(null);
setError("Upload failed — the connection dropped.");
};
xhr.send(file);
}
function inspect() {
if (!sourceKey) return;
resetPreview();
startTransition(async () => {
const result = await inspectRestoreAction(sourceKey, passphrase);
if (result.error) setError(result.error);
setPreview(result.preview);
});
}
function start() {
if (!sourceKey || !preview) return;
setError(null);
startTransition(async () => {
const result = await startRestoreAction(sourceKey, passphrase, replace, confirmSlug);
if (result.error) {
setError(result.error);
return;
}
setNewPassphrase(result.newPassphrase);
setRestoredSlug(preview.slug);
setJobId(result.jobId);
});
}
const blocked = preview ? Boolean(preview.domainConflict || preview.schemaProblem) : false;
const replaceConfirmed = !preview?.slugExists || (replace && confirmSlug === preview.slug);
if (jobId) {
return (
<div className="space-y-4">
{newPassphrase && <PassphraseNotice passphrase={newPassphrase} />}
<JobProgress jobId={jobId}>
{restoredSlug && (
<Link href={`/dashboard/${restoredSlug}`} className="inline-block rounded-md bg-ink px-4 py-2 text-sm font-semibold text-white">
Open {restoredSlug}
</Link>
)}
</JobProgress>
</div>
);
}
return (
<div className="space-y-6">
<fieldset className="space-y-3">
<legend className="text-sm font-semibold">1. Choose the backup</legend>
<div className="flex gap-4 text-sm">
<label className="flex items-center gap-2">
<input
type="radio"
checked={mode === "stored"}
onChange={() => {
setMode("stored");
resetPreview();
}}
disabled={backups.length === 0}
/>
Stored on this server
</label>
<label className="flex items-center gap-2">
<input
type="radio"
checked={mode === "upload"}
onChange={() => {
setMode("upload");
resetPreview();
}}
/>
Upload a .7z
</label>
</div>
{mode === "stored" ? (
<select
value={selected}
onChange={(e) => {
setSelected(e.target.value);
resetPreview();
}}
className="w-full rounded-md border border-line px-3 py-2 font-mono text-xs"
aria-label="Stored backup"
>
{backups.map((b) => (
<option key={b.key} value={b.key}>
{b.slug} — {new Date(b.createdAt).toLocaleString()} — {formatBytes(b.sizeBytes)} — {b.kind}
</option>
))}
</select>
) : (
<div className="space-y-2">
<input
type="file"
accept=".7z"
onChange={(e) => {
const file = e.target.files?.[0];
if (file) upload(file);
}}
className="block w-full text-sm"
/>
<p className="text-xs text-muted">Up to {formatBytes(maxUploadBytes)}. Streamed to the server, not held in memory.</p>
{uploadProgress !== null && (
<div className="h-2 w-full overflow-hidden rounded bg-stone" role="progressbar" aria-valuenow={uploadProgress} aria-valuemin={0} aria-valuemax={100}>
<div className="h-full bg-accent" style={{ width: `${uploadProgress}%` }} />
</div>
)}
{uploadKey && <p className="text-xs text-green-700">Uploaded {uploadName}.</p>}
</div>
)}
</fieldset>
<div className="space-y-2">
<label htmlFor="passphrase" className="block text-sm font-semibold">
2. Passphrase
</label>
<input
id="passphrase"
type="password"
autoComplete="off"
value={passphrase}
onChange={(e) => {
setPassphrase(e.target.value);
resetPreview();
}}
placeholder={hasStoredPassphrase ? "Leave blank to use this server's stored passphrase" : "Required on a fresh server"}
className="w-full rounded-md border border-line px-3 py-2 font-mono text-sm"
/>
<button
type="button"
onClick={inspect}
disabled={isPending || !sourceKey || (!passphrase && !hasStoredPassphrase)}
className="rounded-md bg-ink px-4 py-2 text-sm font-semibold text-white disabled:opacity-60"
>
{isPending && !preview ? "Checking…" : "Check this backup"}
</button>
</div>
{error && (
<p role="alert" className="text-sm font-medium text-danger">
{error}
</p>
)}
{preview && (
<div className="space-y-4">
<h2 className="text-sm font-semibold">3. Review and confirm</h2>
<dl className="grid grid-cols-[auto_1fr] gap-x-4 gap-y-1 rounded-md border border-line bg-surface p-4 text-sm">
<dt className="text-muted">Business</dt>
<dd>{preview.name ?? preview.slug}</dd>
<dt className="text-muted">Slug</dt>
<dd className="font-mono">{preview.slug}</dd>
<dt className="text-muted">Domain</dt>
<dd className="font-mono">{preview.domain}</dd>
<dt className="text-muted">Backup taken</dt>
<dd>{new Date(preview.createdAt).toLocaleString()}</dd>
<dt className="text-muted">Schema</dt>
<dd className="font-mono text-xs">{preview.lastMigration ?? "unknown"}</dd>
</dl>
{preview.domainConflict && (
<p role="alert" className="text-sm font-medium text-danger">
{preview.domainConflict}
</p>
)}
{preview.schemaProblem && (
<p role="alert" className="text-sm font-medium text-danger">
{preview.schemaProblem}
</p>
)}
{preview.slugExists ? (
<div className="space-y-2 rounded-md border border-danger/30 bg-danger/5 p-4 text-sm">
<p className="font-semibold text-danger">
&ldquo;{preview.slug}&rdquo; already exists on this server. Restoring REPLACES its database, media and configuration.
</p>
<p className="text-muted">A safety backup of the current state is taken first.</p>
<label className="flex items-center gap-2">
<input type="checkbox" checked={replace} onChange={(e) => setReplace(e.target.checked)} />
Replace the existing business
</label>
<input
type="text"
value={confirmSlug}
onChange={(e) => setConfirmSlug(e.target.value)}
placeholder={`Type ${preview.slug} to confirm`}
autoComplete="off"
spellCheck={false}
className="w-full rounded-md border border-line px-3 py-2 font-mono text-sm"
/>
</div>
) : (
<p className="text-sm text-muted">No business named &ldquo;{preview.slug}&rdquo; exists here — it will be created from the backup.</p>
)}
<button
type="button"
onClick={start}
disabled={isPending || blocked || !replaceConfirmed}
className="rounded-md bg-danger px-4 py-2 text-sm font-semibold text-white disabled:opacity-40"
>
{isPending ? "Starting…" : preview.slugExists ? "Replace and restore" : "Restore"}
</button>
</div>
)}
</div>
);
}
@@ -16,6 +16,7 @@ import {
resolvePushSettings,
saveEmailSettings,
saveGoogleCalendarSettings,
saveOperatorSettings,
savePushSubject,
saveSuperAdminSettings,
VapidKeysAlreadyExistError,
@@ -23,6 +24,7 @@ import {
type EmailSettingsInput,
type GoogleCalendarSettingsInput,
} from "@/lib/global-settings";
import { sendOperatorEmail } from "@/lib/operator-email";
import { applySuperAdminToAllBusinesses, type SuperAdminRolloutResult } from "@/lib/super-admin-rollout";
import { sendTestSmtpEmail } from "@/lib/test-email";
@@ -359,3 +361,53 @@ export async function reapplyGoogleCalendarSettingsAction(
const rollout = await applyGoogleCalendarSettingsToAllBusinesses(resolved);
return { error: null, rollout };
}
// ── Monitoring & scheduler ───────────────────────────────────────────────
export interface SaveOperatorSettingsState {
error: string | null;
saved: boolean;
}
export async function saveOperatorSettingsAction(
_prevState: SaveOperatorSettingsState,
formData: FormData,
): Promise<SaveOperatorSettingsState> {
const alertEmail = String(formData.get("alertEmail") ?? "").trim();
if (alertEmail && !/^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(alertEmail)) {
return { error: "Enter a valid alert email address, or leave it blank to turn alerts off.", saved: false };
}
const notificationsIntervalSeconds = Number(formData.get("notificationsIntervalSeconds"));
const healthIntervalMinutes = Number(formData.get("healthIntervalMinutes"));
if (!Number.isInteger(notificationsIntervalSeconds) || notificationsIntervalSeconds < 10 || notificationsIntervalSeconds > 3600) {
return { error: "Notification interval must be between 10 and 3600 seconds.", saved: false };
}
if (!Number.isInteger(healthIntervalMinutes) || healthIntervalMinutes < 1 || healthIntervalMinutes > 60) {
return { error: "Health check interval must be between 1 and 60 minutes.", saved: false };
}
try {
await saveOperatorSettings({
alertEmail,
alertOnHealth: formData.get("alertOnHealth") === "on",
alertOnRecovery: formData.get("alertOnRecovery") === "on",
dailyReminder: formData.get("dailyReminder") === "on",
alertOnBackupFailure: formData.get("alertOnBackupFailure") === "on",
notificationsIntervalSeconds,
healthIntervalMinutes,
});
} catch (error) {
return { error: error instanceof Error ? error.message : "Couldn't save settings.", saved: false };
}
return { error: null, saved: true };
}
export interface SendTestAlertState {
result: { ok: boolean; message: string } | null;
}
/** Sends a test alert to the SAVED alert address — so it also proves the Email settings work for alerts. */
export async function sendTestAlertAction(_prevState: SendTestAlertState, _formData: FormData): Promise<SendTestAlertState> {
const result = await sendOperatorEmail("Test alert", "This is a test alert from your NextChair Manager. If you received it, health and backup alerts will reach you.");
return { result };
}
@@ -0,0 +1,167 @@
"use client";
import { useRouter } from "next/navigation";
import { useActionState, useEffect } from "react";
import type { OperatorSettings } from "@/lib/global-settings";
import type { SchedulerStatus } from "@/lib/scheduler-status";
import {
saveOperatorSettingsAction,
sendTestAlertAction,
type SaveOperatorSettingsState,
type SendTestAlertState,
} from "./actions";
const saveInitialState: SaveOperatorSettingsState = { error: null, saved: false };
const testInitialState: SendTestAlertState = { result: null };
function Toggle({ name, label, hint, defaultChecked }: { name: string; label: string; hint?: string; defaultChecked: boolean }) {
return (
<label className="flex items-start gap-2 text-sm">
<input type="checkbox" name={name} defaultChecked={defaultChecked} className="mt-0.5" />
<span>
{label}
{hint && <span className="block text-xs text-muted">{hint}</span>}
</span>
</label>
);
}
export function OperatorSettingsForm({ initial, status }: { initial: OperatorSettings; status: SchedulerStatus }) {
const router = useRouter();
const [saveState, saveAction, isSaving] = useActionState(saveOperatorSettingsAction, saveInitialState);
const [testState, testAction, isTesting] = useActionState(sendTestAlertAction, testInitialState);
useEffect(() => {
if (saveState.saved) router.refresh();
}, [saveState, router]);
return (
<div className="space-y-6">
<div className="flex items-center justify-between">
<h2 className="text-lg font-semibold">Monitoring &amp; scheduler</h2>
{initial.alertEmail ? (
<span className="rounded-full bg-green-100 px-3 py-1 text-xs font-semibold text-green-800">Alerts on</span>
) : (
<span className="rounded-full bg-stone px-3 py-1 text-xs font-semibold text-muted">Alerts off</span>
)}
</div>
<p className="text-sm text-muted">
The Manager calls every business&apos;s notification cron, checks its <span className="font-mono">/api/health</span> and runs scheduled
backups itself. Alerts are emailed with the SMTP settings above; a business must fail two checks in a row before you hear about it.
</p>
<form className="space-y-4 rounded-md border border-line bg-surface p-5">
<div>
<label htmlFor="alertEmail" className="block text-sm font-medium text-muted">
Operator alert email
</label>
<input
id="alertEmail"
name="alertEmail"
type="email"
defaultValue={initial.alertEmail}
placeholder="you@example.com — blank turns alerts off"
className="mt-1 w-full rounded-md border border-line px-3 py-2 text-sm"
/>
</div>
<div className="space-y-2">
<Toggle name="alertOnHealth" label="Email when a business goes degraded or down" defaultChecked={initial.alertOnHealth} />
<Toggle name="alertOnRecovery" label="Email when it recovers" defaultChecked={initial.alertOnRecovery} />
<Toggle name="dailyReminder" label="Remind me daily while it is still bad" defaultChecked={initial.dailyReminder} />
<Toggle name="alertOnBackupFailure" label="Email when a scheduled backup fails" defaultChecked={initial.alertOnBackupFailure} />
</div>
<div className="grid grid-cols-1 gap-4 sm:grid-cols-2">
<div>
<label htmlFor="notificationsIntervalSeconds" className="block text-sm font-medium text-muted">
Notification cron every (seconds)
</label>
<input
id="notificationsIntervalSeconds"
name="notificationsIntervalSeconds"
type="number"
min={10}
max={3600}
required
defaultValue={initial.notificationsIntervalSeconds}
className="mt-1 w-full rounded-md border border-line px-3 py-2 text-sm"
/>
</div>
<div>
<label htmlFor="healthIntervalMinutes" className="block text-sm font-medium text-muted">
Health check every (minutes)
</label>
<input
id="healthIntervalMinutes"
name="healthIntervalMinutes"
type="number"
min={1}
max={60}
required
defaultValue={initial.healthIntervalMinutes}
className="mt-1 w-full rounded-md border border-line px-3 py-2 text-sm"
/>
</div>
</div>
{saveState.error && (
<p role="alert" className="text-sm font-medium text-danger">
{saveState.error}
</p>
)}
{saveState.saved && !saveState.error && <p className="text-sm text-green-700">Saved.</p>}
<div className="flex flex-wrap gap-3 pt-2">
<button
type="submit"
formAction={saveAction}
disabled={isSaving}
className="rounded-md bg-ink px-4 py-2 text-sm font-semibold text-white disabled:opacity-60"
>
{isSaving ? "Saving…" : "Save Changes"}
</button>
<button
type="submit"
formAction={testAction}
disabled={isTesting || !initial.alertEmail}
title={!initial.alertEmail ? "Save an alert email first" : "Sends to the saved address"}
className="ml-auto text-sm font-medium text-muted hover:text-ink disabled:opacity-40"
>
{isTesting ? "Sending…" : "Send test alert"}
</button>
</div>
{testState.result && (
<p role="status" className={`text-sm ${testState.result.ok ? "text-green-700" : "font-medium text-danger"}`}>
{testState.result.message}
</p>
)}
</form>
<div className="rounded-md border border-line bg-surface p-5">
<h3 className="text-sm font-semibold">Scheduler status</h3>
<p className="text-xs text-muted">
Last tick: {status.heartbeatAt ? new Date(status.heartbeatAt).toLocaleString() : "never (scheduler not running?)"}
</p>
{status.rows.length > 0 && (
<table className="mt-3 w-full text-left text-xs">
<thead className="text-muted">
<tr>
<th className="py-1 pr-3 font-medium">Task</th>
<th className="py-1 pr-3 font-medium">Last run</th>
<th className="py-1 font-medium">Result</th>
</tr>
</thead>
<tbody>
{status.rows.map(({ key, state }) => (
<tr key={key} className="border-t border-line">
<td className="py-1 pr-3 font-mono">{key}</td>
<td className="py-1 pr-3">{state.lastFinishedAt ? new Date(state.lastFinishedAt).toLocaleString() : "running / never"}</td>
<td className={`py-1 ${state.lastOk === false ? "text-danger" : ""}`}>{state.lastMessage ?? "—"}</td>
</tr>
))}
</tbody>
</table>
)}
</div>
</div>
);
}
+13 -2
View File
@@ -1,7 +1,15 @@
import type { Metadata } from "next";
import { readEmailSettingsView, readGoogleCalendarSettingsView, readPushSettingsView, readSuperAdminSettings } from "@/lib/global-settings";
import {
readEmailSettingsView,
readGoogleCalendarSettingsView,
readOperatorSettings,
readPushSettingsView,
readSuperAdminSettings,
} from "@/lib/global-settings";
import { readSchedulerStatus } from "@/lib/scheduler-status";
import { EmailSettingsForm } from "./email-settings-form";
import { GoogleCalendarSettingsForm } from "./google-calendar-settings-form";
import { OperatorSettingsForm } from "./operator-settings-form";
import { PushSettingsForm } from "./push-settings-form";
import { SuperAdminSettingsForm } from "./super-admin-settings-form";
@@ -29,11 +37,13 @@ export const dynamic = "force-dynamic";
* branding, locale, monitoring) is a sibling here, not a rewrite.
*/
export default async function SettingsPage() {
const [email, push, superAdmin, googleCalendar] = await Promise.all([
const [email, push, superAdmin, googleCalendar, operator, schedulerStatus] = await Promise.all([
readEmailSettingsView(),
readPushSettingsView(),
readSuperAdminSettings(),
readGoogleCalendarSettingsView(),
readOperatorSettings(),
readSchedulerStatus(),
]);
return (
@@ -46,6 +56,7 @@ export default async function SettingsPage() {
<PushSettingsForm initial={push} />
<SuperAdminSettingsForm initial={superAdmin} />
<GoogleCalendarSettingsForm initial={googleCalendar} />
<OperatorSettingsForm initial={operator} status={schedulerStatus} />
</div>
);
}
+21
View File
@@ -0,0 +1,21 @@
/**
* Runs once when the Manager's server starts (Next.js instrumentation).
* Starts the built-in scheduler — notification cron, daily reviews sync,
* health checks, scheduled backups — as ONE interval guarded by a
* globalThis singleton (see lib/scheduler-start.ts).
*
* Node.js runtime only (the scheduler uses fs/child_process), and skipped
* when STACKS_DIR isn't set or MANAGER_SCHEDULER_DISABLED=1 (e.g. a
* throwaway `next dev` with no stacks, or to hand cron back to the host's
* stacks/infra/scripts/process-cron.sh).
*/
export async function register() {
if (process.env.NEXT_RUNTIME !== "nodejs") return;
if (!process.env.STACKS_DIR) return;
if (process.env.MANAGER_SCHEDULER_DISABLED === "1") {
console.log("[scheduler] disabled by MANAGER_SCHEDULER_DISABLED=1.");
return;
}
const { startScheduler } = await import("./lib/scheduler-start");
await startScheduler();
}
@@ -0,0 +1,83 @@
import { describe, expect, it } from "vitest";
import { buildManifest, checkMigrationCompatibility, ManifestError, parseManifest } from "../backup-manifest";
const valid = () =>
buildManifest({
slug: "acme",
name: "Acme Barbers",
domain: "acme.example.com",
databaseName: "acme",
databaseUser: "acme",
createdAt: "2026-10-17T03:00:00.000Z",
kind: "regular",
lastMigration: "20261017180000_add_business_holidays",
appImage: "nextchair-app:latest",
hasCaddySite: true,
});
describe("parseManifest", () => {
it("round-trips a manifest it built", () => {
const parsed = parseManifest(JSON.stringify(valid()));
expect(parsed).toEqual(valid());
expect(parsed.formatVersion).toBe(1);
});
it("accepts a manifest with no recorded migration", () => {
expect(parseManifest(JSON.stringify({ ...valid(), lastMigration: null })).lastMigration).toBeNull();
});
it("rejects non-JSON and non-objects", () => {
expect(() => parseManifest("not json")).toThrow(ManifestError);
expect(() => parseManifest("[]")).toThrow(ManifestError);
expect(() => parseManifest("null")).toThrow(ManifestError);
});
it("rejects a format newer than this Manager understands", () => {
expect(() => parseManifest(JSON.stringify({ ...valid(), formatVersion: 99 }))).toThrow(/newer than this Manager/);
});
it.each([
["slug", { slug: "Not A Slug" }],
["slug", { slug: "infra" }],
["domain", { domain: 'evil".com' }],
["database name", { databaseName: "acme`; DROP DATABASE x; --" }],
["database user", { databaseUser: "root'@'%" }],
["createdAt", { createdAt: "yesterday" }],
["kind", { kind: "nightly" }],
["migration", { lastMigration: "../../etc" }],
["business name", { name: 'Acme "quoted"' }],
])("rejects an unsafe or invalid %s", (_label, override) => {
expect(() => parseManifest(JSON.stringify({ ...valid(), ...override }))).toThrow(ManifestError);
});
it("rejects a manifest missing required fields", () => {
const { domain: _domain, ...rest } = valid();
expect(() => parseManifest(JSON.stringify(rest))).toThrow(/domain/);
});
});
describe("checkMigrationCompatibility (version guard)", () => {
const image = ["20260912165903_add_staff_selected_by_customer", "20261017180000_add_business_holidays"];
it("allows a backup on the image's newest migration", () => {
expect(checkMigrationCompatibility("20261017180000_add_business_holidays", image)).toEqual({ ok: true });
});
it("allows an older backup (the entrypoint migrates it forward)", () => {
expect(checkMigrationCompatibility("20260912165903_add_staff_selected_by_customer", image)).toEqual({ ok: true });
});
it("refuses a backup whose last migration is newer than the image's", () => {
const result = checkMigrationCompatibility("20271101000000_future_thing", image);
expect(result.ok).toBe(false);
expect(!result.ok && result.reason).toMatch(/newer than the running app image/);
});
it("allows a backup with no recorded migration", () => {
expect(checkMigrationCompatibility(null, image)).toEqual({ ok: true });
});
it("refuses when the image's migration list can't be read", () => {
expect(checkMigrationCompatibility("20261017180000_add_business_holidays", []).ok).toBe(false);
});
});
@@ -0,0 +1,118 @@
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import {
DEFAULT_BACKUP_SETTINGS,
isBackupDue,
lastScheduledOccurrence,
readBackupSettings,
recordBackupAttempt,
recordBackupOutcome,
saveBackupSettings,
validateBackupSettingsInput,
type BackupSettings,
} from "../backup-settings";
// All dates are built from LOCAL components, like the scheduler's own maths, so these hold in any TZ.
const at = (y: number, m: number, d: number, h = 0, min = 0) => new Date(y, m - 1, d, h, min, 0, 0);
const base = (over: Partial<BackupSettings> = {}): BackupSettings => ({ ...DEFAULT_BACKUP_SETTINGS, enabled: true, ...over });
describe("lastScheduledOccurrence", () => {
it("daily: today's time if it has passed, otherwise yesterday's", () => {
const s = { frequency: "daily" as const, time: "03:00", weekday: 0 };
expect(lastScheduledOccurrence(s, at(2026, 10, 17, 14, 30))).toEqual(at(2026, 10, 17, 3, 0));
expect(lastScheduledOccurrence(s, at(2026, 10, 17, 2, 59))).toEqual(at(2026, 10, 16, 3, 0));
expect(lastScheduledOccurrence(s, at(2026, 10, 17, 3, 0))).toEqual(at(2026, 10, 17, 3, 0));
});
it("weekly: the most recent matching weekday at that time", () => {
// 2026-10-17 is a Saturday (6). Weekly on Monday (1) 04:00 → Monday 12th.
const s = { frequency: "weekly" as const, time: "04:00", weekday: 1 };
expect(lastScheduledOccurrence(s, at(2026, 10, 17, 12, 0))).toEqual(at(2026, 10, 12, 4, 0));
// On Monday before 04:00 it is still the previous Monday.
expect(lastScheduledOccurrence(s, at(2026, 10, 19, 3, 0))).toEqual(at(2026, 10, 12, 4, 0));
expect(lastScheduledOccurrence(s, at(2026, 10, 19, 4, 1))).toEqual(at(2026, 10, 19, 4, 0));
});
});
describe("isBackupDue", () => {
it("is never due when disabled", () => {
expect(isBackupDue(base({ enabled: false }), at(2026, 10, 17, 12))).toBe(false);
});
it("does not fire the moment it is enabled — it waits for the next scheduled time", () => {
const enabledAt = at(2026, 10, 17, 14, 0).toISOString();
const s = base({ time: "03:00", enabledAt });
expect(isBackupDue(s, at(2026, 10, 17, 14, 1))).toBe(false);
expect(isBackupDue(s, at(2026, 10, 18, 3, 0))).toBe(true);
});
it("is due once a scheduled time has passed since the last attempt, and not again until the next", () => {
const s = base({ time: "03:00", lastAttemptAt: at(2026, 10, 16, 3, 0).toISOString(), enabledAt: at(2026, 10, 1).toISOString() });
expect(isBackupDue(s, at(2026, 10, 17, 2, 59))).toBe(false);
expect(isBackupDue(s, at(2026, 10, 17, 3, 0))).toBe(true);
const after = { ...s, lastAttemptAt: at(2026, 10, 17, 3, 0, ).toISOString() };
expect(isBackupDue(after, at(2026, 10, 17, 23, 0))).toBe(false);
expect(isBackupDue(after, at(2026, 10, 18, 3, 0))).toBe(true);
});
it("catches up a run missed while the Manager was down", () => {
const s = base({ time: "03:00", lastAttemptAt: at(2026, 10, 10, 3, 0).toISOString(), enabledAt: at(2026, 10, 1).toISOString() });
expect(isBackupDue(s, at(2026, 10, 17, 9, 0))).toBe(true);
});
it("weekly: due only after that weekday's time", () => {
const s = base({ frequency: "weekly", weekday: 1, time: "04:00", lastAttemptAt: at(2026, 10, 12, 4, 0).toISOString(), enabledAt: at(2026, 9, 1).toISOString() });
expect(isBackupDue(s, at(2026, 10, 18, 12))).toBe(false); // Sunday
expect(isBackupDue(s, at(2026, 10, 19, 4, 0))).toBe(true); // next Monday 04:00
});
});
describe("settings store", () => {
let stacksDir: string;
const originalStacksDir = process.env.STACKS_DIR;
beforeEach(async () => {
stacksDir = await mkdtemp(path.join(tmpdir(), "nc-manager-backup-settings-"));
process.env.STACKS_DIR = stacksDir;
});
afterEach(async () => {
if (originalStacksDir === undefined) delete process.env.STACKS_DIR;
else process.env.STACKS_DIR = originalStacksDir;
await rm(stacksDir, { recursive: true, force: true });
});
it("defaults to disabled, daily, 03:00, keep 2", async () => {
expect(await readBackupSettings("acme")).toMatchObject({ enabled: false, frequency: "daily", time: "03:00", keep: 2 });
});
it("is keyed by slug and records enabledAt only on the off→on transition", async () => {
const t1 = new Date("2026-10-17T10:00:00Z");
await saveBackupSettings("acme", { enabled: true, frequency: "weekly", time: "05:30", weekday: 3, keep: 4 }, t1);
await saveBackupSettings("zeta", { enabled: false, frequency: "daily", time: "01:00", weekday: 0, keep: 2 });
const acme = await readBackupSettings("acme");
expect(acme).toMatchObject({ enabled: true, frequency: "weekly", time: "05:30", weekday: 3, keep: 4, enabledAt: t1.toISOString() });
expect((await readBackupSettings("zeta")).enabled).toBe(false);
await saveBackupSettings("acme", { enabled: true, frequency: "daily", time: "05:30", weekday: 0, keep: 4 }, new Date("2026-10-18T10:00:00Z"));
expect((await readBackupSettings("acme")).enabledAt).toBe(t1.toISOString());
});
it("records attempts and outcomes without touching the schedule", async () => {
await saveBackupSettings("acme", { enabled: true, frequency: "daily", time: "02:00", weekday: 0, keep: 2 });
await recordBackupAttempt("acme", new Date("2026-10-17T02:00:00Z"));
await recordBackupOutcome("acme", { ok: false, message: "disk full", at: new Date("2026-10-17T02:05:00Z") });
expect(await readBackupSettings("acme")).toMatchObject({ time: "02:00", lastStatus: "failed", lastMessage: "disk full", lastSuccessAt: null });
await recordBackupOutcome("acme", { ok: true, message: "ok", at: new Date("2026-10-18T02:05:00Z") });
expect((await readBackupSettings("acme")).lastSuccessAt).toBe("2026-10-18T02:05:00.000Z");
});
it("validates input", () => {
const ok = { enabled: true, frequency: "daily" as const, time: "03:00", weekday: 0, keep: 2 };
expect(validateBackupSettingsInput(ok)).toBeNull();
expect(validateBackupSettingsInput({ ...ok, time: "25:00" })).toMatch(/HH:MM/);
expect(validateBackupSettingsInput({ ...ok, keep: 0 })).toMatch(/Keep between/);
expect(validateBackupSettingsInput({ ...ok, keep: 99 })).toMatch(/Keep between/);
expect(validateBackupSettingsInput({ ...ok, weekday: 9 })).toMatch(/day/);
});
});
@@ -0,0 +1,106 @@
import { mkdtemp, mkdir, readdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { backupFileName, backupFilePath, listAllBackups, listBackups, parseBackupFileName, pruneBackups } from "../backup-store";
describe("backup file naming", () => {
it("round-trips slug, kind and UTC timestamp", () => {
const name = backupFileName("acme-cuts", "regular", new Date("2026-10-17T03:04:05.678Z"));
expect(name).toBe("acme-cuts__20261017T030405Z__regular.7z");
expect(parseBackupFileName(name)).toEqual({ slug: "acme-cuts", kind: "regular", createdAt: "2026-10-17T03:04:05Z" });
});
it("rejects names this module did not produce", () => {
expect(parseBackupFileName("acme__20261017T030405Z__regular.7z.partial")).toBeNull();
expect(parseBackupFileName("acme__20261017T030405Z__weird.7z")).toBeNull();
expect(parseBackupFileName("../etc/passwd")).toBeNull();
});
it("refuses to resolve a path for a mismatched slug or a traversal attempt", () => {
process.env.STACKS_DIR = "/stacks";
expect(() => backupFilePath("acme", "other__20261017T030405Z__regular.7z")).toThrow(/Invalid backup reference/);
expect(() => backupFilePath("acme", "../acme__20261017T030405Z__regular.7z")).toThrow(/Invalid backup reference/);
expect(() => backupFilePath("../x", "acme__20261017T030405Z__regular.7z")).toThrow(/Invalid backup reference/);
expect(backupFilePath("acme", "acme__20261017T030405Z__regular.7z")).toBe("/stacks/backups/acme/acme__20261017T030405Z__regular.7z");
});
});
describe("retention", () => {
let stacksDir: string;
const originalStacksDir = process.env.STACKS_DIR;
beforeEach(async () => {
stacksDir = await mkdtemp(path.join(tmpdir(), "nc-manager-backup-store-"));
process.env.STACKS_DIR = stacksDir;
});
afterEach(async () => {
if (originalStacksDir === undefined) delete process.env.STACKS_DIR;
else process.env.STACKS_DIR = originalStacksDir;
await rm(stacksDir, { recursive: true, force: true });
});
async function put(slug: string, kind: "regular" | "safety" | "final", iso: string) {
const dir = path.join(stacksDir, "backups", slug);
await mkdir(dir, { recursive: true });
const name = backupFileName(slug, kind, new Date(iso));
await writeFile(path.join(dir, name), "x");
return name;
}
it("keeps only the newest 2 regular backups and deletes the rest", async () => {
const oldest = await put("acme", "regular", "2026-10-10T03:00:00Z");
const middle = await put("acme", "regular", "2026-10-11T03:00:00Z");
const newest = await put("acme", "regular", "2026-10-12T03:00:00Z");
const removed = await pruneBackups("acme", "regular", 2);
expect(removed).toEqual([oldest]);
expect((await listBackups("acme")).map((b) => b.fileName)).toEqual([newest, middle]);
});
it("does nothing when at or below the limit", async () => {
await put("acme", "regular", "2026-10-10T03:00:00Z");
await put("acme", "regular", "2026-10-11T03:00:00Z");
expect(await pruneBackups("acme", "regular", 2)).toEqual([]);
});
it("honours a per-business keep count", async () => {
for (const day of ["10", "11", "12", "13"]) await put("acme", "regular", `2026-10-${day}T03:00:00Z`);
await pruneBackups("acme", "regular", 3);
expect(await listBackups("acme")).toHaveLength(3);
});
it("never lets regular pruning touch safety or final backups, or another business", async () => {
const safety = await put("acme", "safety", "2026-01-01T00:00:00Z");
const final = await put("acme", "final", "2026-01-02T00:00:00Z");
const other = await put("zeta", "regular", "2026-01-03T00:00:00Z");
for (const day of ["10", "11", "12"]) await put("acme", "regular", `2026-10-${day}T03:00:00Z`);
await pruneBackups("acme", "regular", 1);
const acme = (await listBackups("acme")).map((b) => b.fileName);
expect(acme).toContain(safety);
expect(acme).toContain(final);
expect(acme.filter((n) => n.endsWith("__regular.7z"))).toHaveLength(1);
expect((await listBackups("zeta")).map((b) => b.fileName)).toEqual([other]);
});
it("never prunes final backups", async () => {
for (const day of ["10", "11", "12"]) await put("acme", "final", `2026-10-${day}T03:00:00Z`);
expect(await pruneBackups("acme", "final", 1)).toEqual([]);
expect(await listBackups("acme")).toHaveLength(3);
});
it("ignores .partial files and the uploads folder when listing", async () => {
await put("acme", "regular", "2026-10-10T03:00:00Z");
await writeFile(path.join(stacksDir, "backups", "acme", "acme__20261011T030000Z__regular.7z.partial"), "x");
await mkdir(path.join(stacksDir, "backups", "_uploads"), { recursive: true });
await writeFile(path.join(stacksDir, "backups", "_uploads", "abc.7z"), "x");
expect(await listBackups("acme")).toHaveLength(1);
expect((await listAllBackups()).map((b) => b.slug)).toEqual(["acme"]);
expect(await readdir(path.join(stacksDir, "backups", "acme"))).toHaveLength(2);
});
});
+201
View File
@@ -0,0 +1,201 @@
import { mkdir, mkdtemp, readdir, readFile, rm, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { CommandSpec } from "../process";
const containerExistsMock = vi.fn();
vi.mock("@/lib/docker", () => ({ containerExists: (...a: unknown[]) => containerExistsMock(...a) }));
const runToFileMock = vi.fn();
const runCaptureMock = vi.fn();
vi.mock("@/lib/process", () => ({
runToFile: (...a: unknown[]) => runToFileMock(...a),
runCapture: (...a: unknown[]) => runCaptureMock(...a),
}));
const sevenZipCreateMock = vi.fn();
const sevenZipTestMock = vi.fn();
vi.mock("@/lib/sevenzip", () => ({
sevenZipCreate: (...a: unknown[]) => sevenZipCreateMock(...a),
sevenZipTest: (...a: unknown[]) => sevenZipTestMock(...a),
}));
const readBackupPassphraseMock = vi.fn();
vi.mock("@/lib/global-settings", () => ({ readBackupPassphrase: () => readBackupPassphraseMock() }));
import { performBackup } from "../backup";
import { backupFileName, listBackups } from "../backup-store";
import { saveBackupSettings } from "../backup-settings";
import { parseManifest } from "../backup-manifest";
const PASSPHRASE = "test-passphrase-123";
const ENV = [
'BUSINESS_NAME="Acme Barbers"',
'DOMAIN="acme.example.com"',
'DATABASE_NAME="acme"',
'DATABASE_USER="acme"',
'DATABASE_PASSWORD="abc123"',
].join("\n");
describe("performBackup", () => {
let stacksDir: string;
const originalStacksDir = process.env.STACKS_DIR;
const originalRoot = process.env.MYSQL_ROOT_PASSWORD;
/** What the (mocked) 7zz saw in the staging dir at archive time. */
let staged: { files: string[]; manifest: string };
beforeEach(async () => {
stacksDir = await mkdtemp(path.join(tmpdir(), "nc-manager-backup-"));
process.env.STACKS_DIR = stacksDir;
process.env.MYSQL_ROOT_PASSWORD = "root-pw";
await mkdir(path.join(stacksDir, "acme"), { recursive: true });
await writeFile(path.join(stacksDir, "acme", "compose.yaml"), "name: next-chair-acme\n");
await writeFile(path.join(stacksDir, "acme", ".env"), ENV);
await mkdir(path.join(stacksDir, "infra", "sites"), { recursive: true });
await writeFile(path.join(stacksDir, "infra", "sites", "acme.caddy"), "acme.example.com {\n}\n");
containerExistsMock.mockReset().mockResolvedValue(true);
readBackupPassphraseMock.mockReset().mockResolvedValue(PASSPHRASE);
runCaptureMock.mockReset().mockResolvedValue({ stdout: "20261017180000_add_business_holidays\n", stderr: "" });
runToFileMock.mockReset().mockImplementation(async (spec: CommandSpec, file: string) => {
const isDump = spec.args.includes("mysqldump");
await writeFile(file, isDump ? "-- dump\nCREATE TABLE t (id int);\n-- Dump completed on 2026-10-17\n" : "TAR");
});
sevenZipTestMock.mockReset().mockResolvedValue(undefined);
sevenZipCreateMock.mockReset().mockImplementation(async (p: { archivePath: string; cwd: string }) => {
const files = (await readdir(p.cwd, { recursive: true })).map(String).sort();
staged = { files, manifest: await readFile(path.join(p.cwd, "manifest.json"), "utf8") };
await writeFile(p.archivePath, "7Z-ARCHIVE");
});
});
afterEach(async () => {
if (originalStacksDir === undefined) delete process.env.STACKS_DIR;
else process.env.STACKS_DIR = originalStacksDir;
if (originalRoot === undefined) delete process.env.MYSQL_ROOT_PASSWORD;
else process.env.MYSQL_ROOT_PASSWORD = originalRoot;
await rm(stacksDir, { recursive: true, force: true });
});
const logs = () => {
const lines: string[] = [];
return { lines, ctx: { log: (l: string) => lines.push(l) } };
};
it("streams the dump and the media to files, packages them with a manifest, verifies, then renames into place", async () => {
const { ctx, lines } = logs();
const now = new Date("2026-10-17T03:00:00Z");
const result = await performBackup("acme", "regular", ctx, now);
// 1. mysqldump → db.sql, 2. docker cp → media.tar (both streamed, never captured in memory)
const [dumpSpec, dumpFile] = runToFileMock.mock.calls[0] as [CommandSpec, string];
expect(dumpSpec.args).toEqual(expect.arrayContaining(["mysqldump", "--single-transaction", "--routines", "--triggers", "acme"]));
expect(path.basename(dumpFile)).toBe("db.sql");
const [mediaSpec, mediaFile] = runToFileMock.mock.calls[1] as [CommandSpec, string];
expect(mediaSpec.args).toEqual(["cp", "next-chair-acme:/app/storage/media", "-"]);
expect(path.basename(mediaFile)).toBe("media.tar");
// 3. everything that goes into the archive
expect(staged.files).toEqual(["config", "config/.env", "config/compose.yaml", "config/site.caddy", "db.sql", "manifest.json", "media.tar"]);
const manifest = parseManifest(staged.manifest);
expect(manifest).toMatchObject({
slug: "acme", name: "Acme Barbers", domain: "acme.example.com", databaseName: "acme", databaseUser: "acme",
kind: "regular", lastMigration: "20261017180000_add_business_holidays", hasCaddySite: true, createdAt: now.toISOString(),
});
expect(staged.manifest).not.toContain("abc123"); // the DB password lives only in config/.env inside the encrypted archive
// 4. encrypted with the stored passphrase, written as .partial first
const create = sevenZipCreateMock.mock.calls[0]![0] as { archivePath: string; passphrase: string; entries: string[] };
expect(create.passphrase).toBe(PASSPHRASE);
expect(create.archivePath.endsWith(`${backupFileName("acme", "regular", now)}.partial`)).toBe(true);
expect(create.entries).toEqual(["manifest.json", "db.sql", "media.tar", "config"]);
// 5. verified (7zz t) the .partial, THEN renamed
expect(sevenZipTestMock).toHaveBeenCalledWith(create.archivePath, PASSPHRASE);
const finalPath = create.archivePath.replace(/\.partial$/, "");
expect((await stat(finalPath)).isFile()).toBe(true);
expect((await stat(finalPath)).mode & 0o777).toBe(0o600);
await expect(stat(create.archivePath)).rejects.toThrow();
expect(result).toMatchObject({ slug: "acme", kind: "regular", fileName: backupFileName("acme", "regular", now) });
// no staging leftovers, and nothing secret in the log
expect(await readdir(path.join(stacksDir, "manager", "tmp"))).toEqual([]);
expect(lines.join("\n")).not.toContain(PASSPHRASE);
expect(lines.join("\n")).not.toContain("abc123");
});
it("keeps only the newest 2 backups after a successful run", async () => {
const dir = path.join(stacksDir, "backups", "acme");
await mkdir(dir, { recursive: true });
for (const iso of ["2026-10-10T03:00:00Z", "2026-10-11T03:00:00Z", "2026-10-12T03:00:00Z"]) {
await writeFile(path.join(dir, backupFileName("acme", "regular", new Date(iso))), "old");
}
const result = await performBackup("acme", "regular", logs().ctx, new Date("2026-10-13T03:00:00Z"));
expect((await listBackups("acme")).map((b) => b.createdAt)).toEqual(["2026-10-13T03:00:00Z", "2026-10-12T03:00:00Z"]);
expect(result.pruned).toHaveLength(2);
});
it("uses the business's configured keep count", async () => {
await saveBackupSettings("acme", { enabled: true, frequency: "daily", time: "03:00", weekday: 0, keep: 3 });
const dir = path.join(stacksDir, "backups", "acme");
await mkdir(dir, { recursive: true });
for (const day of ["10", "11", "12"]) await writeFile(path.join(dir, backupFileName("acme", "regular", new Date(`2026-10-${day}T03:00:00Z`))), "old");
await performBackup("acme", "regular", logs().ctx, new Date("2026-10-13T03:00:00Z"));
expect(await listBackups("acme")).toHaveLength(3);
});
it("a failed verification leaves NO new backup behind and does not prune the good ones", async () => {
const dir = path.join(stacksDir, "backups", "acme");
await mkdir(dir, { recursive: true });
const good = [backupFileName("acme", "regular", new Date("2026-10-11T03:00:00Z")), backupFileName("acme", "regular", new Date("2026-10-12T03:00:00Z"))];
for (const name of good) await writeFile(path.join(dir, name), "good");
sevenZipTestMock.mockRejectedValue(new Error("7zz exited with code 2: Data Error"));
await expect(performBackup("acme", "regular", logs().ctx, new Date("2026-10-13T03:00:00Z"))).rejects.toThrow(/Data Error/);
expect((await readdir(dir)).sort()).toEqual([...good].sort());
expect(await readdir(path.join(stacksDir, "manager", "tmp"))).toEqual([]);
});
it("refuses an incomplete dump (no completion marker)", async () => {
runToFileMock.mockImplementation(async (_spec: CommandSpec, file: string) => writeFile(file, "-- dump\nCREATE TABLE t (id int);\n"));
await expect(performBackup("acme", "regular", logs().ctx)).rejects.toThrow(/incomplete/);
expect(sevenZipCreateMock).not.toHaveBeenCalled();
});
it("reads media straight from the volume when the container is gone (a stopped stack is compose-down'd)", async () => {
containerExistsMock.mockResolvedValue(false);
await performBackup("acme", "regular", logs().ctx);
const [mediaSpec] = runToFileMock.mock.calls[1] as [CommandSpec];
expect(mediaSpec.args).toContain("next-chair-media-acme:/app/storage/media:ro");
});
it("safety backups prune only other safety backups and never the regular ones", async () => {
const dir = path.join(stacksDir, "backups", "acme");
await mkdir(dir, { recursive: true });
const regular = backupFileName("acme", "regular", new Date("2026-10-01T03:00:00Z"));
await writeFile(path.join(dir, regular), "r");
for (const day of ["02", "03", "04"]) await writeFile(path.join(dir, backupFileName("acme", "safety", new Date(`2026-10-${day}T03:00:00Z`))), "s");
await performBackup("acme", "safety", logs().ctx, new Date("2026-10-05T03:00:00Z"));
const names = (await readdir(dir)).sort();
expect(names).toContain(regular);
expect(names.filter((n) => n.includes("__safety"))).toHaveLength(2);
});
it("fails clearly when no passphrase exists yet", async () => {
readBackupPassphraseMock.mockResolvedValue(null);
await expect(performBackup("acme", "regular", logs().ctx)).rejects.toThrow(/No backup passphrase/);
expect(runToFileMock).not.toHaveBeenCalled();
});
it("emits job steps in order when given a step wrapper", async () => {
const order: string[] = [];
await performBackup("acme", "regular", { log: () => undefined, step: async (id, fn) => (order.push(id), fn()) });
expect(order).toEqual(["dump", "media", "config", "archive", "verify", "finish"]);
});
});
@@ -0,0 +1,124 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import {
containerExistsCommand,
imageMigrationsCommand,
mediaCopyOutCommand,
mediaRestoreCommand,
mediaVolumeTarCommand,
mysqlImportCommand,
mysqlQueryCommand,
mysqldumpCommand,
} from "../docker-commands";
import { sevenZipCreateCommand, sevenZipExtractToStdoutCommand, sevenZipTestCommand } from "../sevenzip";
const ROOT_PASSWORD = "root-pw-for-tests";
describe("command construction", () => {
const saved = { pw: process.env.MYSQL_ROOT_PASSWORD, image: process.env.NEXTCHAIR_IMAGE, mysql: process.env.MYSQL_CONTAINER_NAME };
beforeEach(() => {
process.env.MYSQL_ROOT_PASSWORD = ROOT_PASSWORD;
delete process.env.NEXTCHAIR_IMAGE;
delete process.env.MYSQL_CONTAINER_NAME;
});
afterEach(() => {
for (const [key, value] of [["MYSQL_ROOT_PASSWORD", saved.pw], ["NEXTCHAIR_IMAGE", saved.image], ["MYSQL_CONTAINER_NAME", saved.mysql]] as const) {
if (value === undefined) delete process.env[key];
else process.env[key] = value;
}
});
it("mysqldump: single-transaction + routines + triggers, as an argument array, password only in env", () => {
const spec = mysqldumpCommand("acme");
expect(spec.command).toBe("docker");
expect(spec.args).toEqual([
"exec", "-e", "MYSQL_PWD", "next-chair-mysql", "mysqldump", "-uroot",
"--single-transaction", "--routines", "--triggers", "--no-tablespaces", "--hex-blob",
"--default-character-set=utf8mb4", "acme",
]);
expect(spec.env).toEqual({ MYSQL_PWD: ROOT_PASSWORD });
expect(spec.args.join(" ")).not.toContain(ROOT_PASSWORD);
});
it("mysql import reads the dump on stdin (-i) and also keeps the password out of argv", () => {
const spec = mysqlImportCommand("acme");
expect(spec.args).toEqual(["exec", "-i", "-e", "MYSQL_PWD", "next-chair-mysql", "mysql", "-uroot", "--default-character-set=utf8mb4", "acme"]);
expect(spec.args.join(" ")).not.toContain(ROOT_PASSWORD);
expect(spec.env?.MYSQL_PWD).toBe(ROOT_PASSWORD);
});
it("mysql query passes the SQL as a single argument (never through a shell)", () => {
const spec = mysqlQueryCommand("acme", "SELECT 1; SELECT 2");
expect(spec.args.at(-2)).toBe("-e");
expect(spec.args.at(-1)).toBe("SELECT 1; SELECT 2");
});
it("honours MYSQL_CONTAINER_NAME and NEXTCHAIR_IMAGE overrides", () => {
process.env.MYSQL_CONTAINER_NAME = "custom-mysql";
process.env.NEXTCHAIR_IMAGE = "registry.example/app:1.2";
expect(mysqldumpCommand("acme").args).toContain("custom-mysql");
expect(imageMigrationsCommand().args).toEqual(["run", "--rm", "--entrypoint", "ls", "registry.example/app:1.2", "/app/prisma/migrations"]);
});
it("media is streamed out with docker cp to stdout, or read from the volume when the container is gone", () => {
expect(mediaCopyOutCommand("next-chair-acme").args).toEqual(["cp", "next-chair-acme:/app/storage/media", "-"]);
expect(mediaVolumeTarCommand("acme").args).toEqual([
"run", "--rm", "-v", "next-chair-media-acme:/app/storage/media:ro", "--entrypoint", "tar", "nextchair-app:latest",
"-c", "-C", "/app/storage", "media",
]);
});
it("media restore unpacks stdin as root into the volume, then chowns to 1001:1001", () => {
const spec = mediaRestoreCommand("acme");
expect(spec.args.slice(0, 9)).toEqual(["run", "--rm", "-i", "--user", "0", "-v", "next-chair-media-acme:/app/storage/media", "--entrypoint", "sh"]);
expect(spec.args.at(-1)).toBe("tar -x -C /app/storage && chown -R 1001:1001 /app/storage/media");
});
it("container existence check", () => {
expect(containerExistsCommand("next-chair-acme").args).toEqual(["container", "inspect", "--format", "{{.Id}}", "next-chair-acme"]);
});
it("requires MYSQL_ROOT_PASSWORD", () => {
delete process.env.MYSQL_ROOT_PASSWORD;
expect(() => mysqldumpCommand("acme")).toThrow(/MYSQL_ROOT_PASSWORD/);
});
});
describe("7zz command construction", () => {
const passphrase = "correct-horse-battery";
it("create: 7z, encrypted headers, passphrase, listed entries, run from the staging dir", () => {
const spec = sevenZipCreateCommand({
archivePath: "/stacks/backups/acme/x.7z.partial",
cwd: "/stacks/manager/tmp/s1",
entries: ["manifest.json", "db.sql", "media.tar", "config"],
passphrase,
});
expect(spec.command).toBe("7zz");
expect(spec.cwd).toBe("/stacks/manager/tmp/s1");
expect(spec.args).toEqual([
"a", "-t7z", "-mx=3", "-mhe=on", `-p${passphrase}`, "-y", "-bd",
"/stacks/backups/acme/x.7z.partial", "manifest.json", "db.sql", "media.tar", "config",
]);
});
it("test and stdout-extract commands", () => {
expect(sevenZipTestCommand("/a.7z", passphrase).args).toEqual(["t", "-t7z", `-p${passphrase}`, "-y", "-bd", "/a.7z"]);
expect(sevenZipExtractToStdoutCommand("/a.7z", passphrase, "db.sql").args).toEqual([
"x", "-so", "-t7z", `-p${passphrase}`, "-y", "-bd", "-bso0", "-bsp0", "/a.7z", "db.sql",
]);
});
it("refuses a short passphrase (an empty -p would hang on a prompt)", () => {
expect(() => sevenZipTestCommand("/a.7z", "")).toThrow(/at least 8/);
expect(() => sevenZipTestCommand("/a.7z", "short")).toThrow(/at least 8/);
});
it("SEVENZIP_BIN overrides the binary", () => {
process.env.SEVENZIP_BIN = "/opt/7zz";
try {
expect(sevenZipTestCommand("/a.7z", passphrase).command).toBe("/opt/7zz");
} finally {
delete process.env.SEVENZIP_BIN;
}
});
});
@@ -0,0 +1,245 @@
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { DEFAULT_OPERATOR_SETTINGS } from "../global-settings";
import { classifyHealthBody, waitForHealth, type HealthSnapshot } from "../health";
import { applyObservation, buildAlertEmail, REMINDER_INTERVAL_MS, runHealthCheck, setIntentionallyStopped, type HealthRecord } from "../health-watcher";
const t0 = new Date("2026-10-17T12:00:00Z");
const at = (minutes: number) => new Date(t0.getTime() + minutes * 60_000);
const OK = { verdict: "ok" as const, reasons: [] };
const DOWN = { verdict: "down" as const, reasons: ["couldn't reach the app"] };
const DEGRADED = { verdict: "degraded" as const, reasons: ["12 notifications stale"] };
const opts = { dailyReminder: true };
describe("health state machine (pure)", () => {
it("one failed check does NOT alert; the second consecutive one does", () => {
const first = applyObservation("acme", undefined, DOWN, at(0), opts);
expect(first.alert).toBeNull();
expect(first.next.status).toBe("ok");
expect(first.next.pending).toEqual({ verdict: "down", count: 1 });
const second = applyObservation("acme", first.next, DOWN, at(3), opts);
expect(second.alert).toMatchObject({ type: "problem", from: "ok", to: "down" });
expect(second.next.status).toBe("down");
expect(second.next.pending).toBeNull();
});
it("a single blip in between resets the count", () => {
const a = applyObservation("acme", undefined, DOWN, at(0), opts);
const b = applyObservation("acme", a.next, OK, at(3), opts);
const c = applyObservation("acme", b.next, DOWN, at(6), opts);
expect(c.alert).toBeNull();
expect(c.next.pending?.count).toBe(1);
});
it("alerts on a change between bad states and on recovery", () => {
let rec: HealthRecord | undefined;
const run = (obs: typeof OK | typeof DOWN | typeof DEGRADED, min: number) => {
const r = applyObservation("acme", rec, obs, at(min), opts);
rec = r.next;
return r.alert;
};
run(DEGRADED, 0);
expect(run(DEGRADED, 3)).toMatchObject({ type: "problem", to: "degraded" });
run(DOWN, 6);
expect(run(DOWN, 9)).toMatchObject({ type: "changed", from: "degraded", to: "down" });
// Recovery reports when the bad period began (the down state started at minute 9).
expect(run(OK, 12)).toMatchObject({ type: "recovered", from: "down", to: "ok", since: at(9).toISOString() });
expect(run(OK, 15)).toBeNull();
});
it("sends a reminder once a day while still bad — not before, and not for ok", () => {
const bad = applyObservation("acme", applyObservation("acme", undefined, DOWN, at(0), opts).next, DOWN, at(3), opts);
const alerted: HealthRecord = { ...bad.next, lastAlertAt: at(3).toISOString() };
const early = applyObservation("acme", alerted, DOWN, new Date(at(3).getTime() + REMINDER_INTERVAL_MS - 60_000), opts);
expect(early.alert).toBeNull();
const due = applyObservation("acme", alerted, DOWN, new Date(at(3).getTime() + REMINDER_INTERVAL_MS), opts);
expect(due.alert).toMatchObject({ type: "reminder", to: "down" });
const off = applyObservation("acme", alerted, DOWN, new Date(at(3).getTime() + REMINDER_INTERVAL_MS), { dailyReminder: false });
expect(off.alert).toBeNull();
const healthy = applyObservation("acme", undefined, OK, new Date(t0.getTime() + 3 * REMINDER_INTERVAL_MS), opts);
expect(healthy.alert).toBeNull();
});
it("alert emails say what happened and never need a secret", () => {
const problem = buildAlertEmail({ type: "problem", slug: "acme", from: "ok", to: "down", reasons: ["connection refused"], since: at(3).toISOString(), raisedAt: at(3).toISOString() }, "Acme Barbers", "acme.example.com");
expect(problem.subject).toBe("Acme Barbers (acme.example.com) is DOWN");
expect(problem.text).toContain("connection refused");
const recovered = buildAlertEmail({ type: "recovered", slug: "acme", from: "down", to: "ok", reasons: [], since: at(9).toISOString(), raisedAt: at(9).toISOString() }, "Acme Barbers", null);
expect(recovered.subject).toBe("Acme Barbers has recovered");
expect(recovered.text).toContain(`was down since ${at(9).toISOString()}`);
});
});
describe("classifyHealthBody — consuming the app's /api/health", () => {
it("treats today's body (no queue/cron fields) as healthy by reachability", () => {
const s = classifyHealthBody(200, { status: "healthy", database: "reachable", installed: true, installedAt: "2026-01-01T00:00:00.000Z" });
expect(s).toMatchObject({ verdict: "ok", reasons: [], installed: true, queue: null, lastCronRunAt: null });
});
it("a non-200 or non-JSON response is down", () => {
expect(classifyHealthBody(503, { status: "unhealthy" }).verdict).toBe("down");
expect(classifyHealthBody(200, null).verdict).toBe("down");
});
it("consumes the planned queue/cron fields and degraded status with reasons", () => {
const s = classifyHealthBody(200, {
status: "degraded",
installed: true,
degradedReasons: ["12 notifications pending for over 10 minutes", "cron hasn't run since 11:00"],
queue: { stalePendingCount: 12, oldestPendingAgeSeconds: 900, stuckProcessingCount: 1, failedLast24h: 3 },
cron: { lastCronRunAt: "2026-10-17T11:00:00.000Z" },
});
expect(s.verdict).toBe("degraded");
expect(s.reasons).toHaveLength(2);
expect(s.queue).toEqual({ stalePendingCount: 12, oldestPendingAgeSeconds: 900, stuckProcessingCount: 1, failedLast24h: 3 });
expect(s.lastCronRunAt).toBe("2026-10-17T11:00:00.000Z");
});
it("status ok with the new fields present is still ok", () => {
expect(classifyHealthBody(200, { status: "ok", installed: true, queue: { stalePendingCount: 0 }, cron: {} }).verdict).toBe("ok");
});
it("degraded with no reasons still gets a reason", () => {
expect(classifyHealthBody(200, { status: "degraded" }).reasons).toEqual(["the app reports itself degraded"]);
});
});
describe("waitForHealth", () => {
const snap = (over: Partial<HealthSnapshot>): HealthSnapshot => ({ verdict: "ok", reasons: [], httpStatus: 200, installed: true, installedAt: null, lastCronRunAt: null, queue: null, ...over });
it("returns once the app answers 200", async () => {
const probe = vi.fn().mockResolvedValueOnce(snap({ httpStatus: null, verdict: "down" })).mockResolvedValueOnce(snap({}));
await expect(waitForHealth("acme", { probe, sleep: async () => undefined })).resolves.toMatchObject({ httpStatus: 200 });
expect(probe).toHaveBeenCalledTimes(2);
});
it("restore mode insists on installed:true — a reachable but un-installed app times out with a wizard warning", async () => {
const probe = vi.fn().mockResolvedValue(snap({ installed: false }));
await expect(waitForHealth("acme", { probe, sleep: async () => undefined, timeoutMs: 0, requireInstalled: true })).rejects.toThrow(/setup wizard/);
});
it("times out with the reasons when never reachable", async () => {
const probe = vi.fn().mockResolvedValue(snap({ httpStatus: null, verdict: "down", reasons: ["connection refused"] }));
await expect(waitForHealth("acme", { probe, sleep: async () => undefined, timeoutMs: 0 })).rejects.toThrow(/did not become healthy.*connection refused/);
});
});
describe("runHealthCheck (state file + email wiring)", () => {
let stacksDir: string;
const originalStacksDir = process.env.STACKS_DIR;
beforeEach(async () => {
stacksDir = await mkdtemp(path.join(tmpdir(), "nc-manager-health-"));
process.env.STACKS_DIR = stacksDir;
});
afterEach(async () => {
if (originalStacksDir === undefined) delete process.env.STACKS_DIR;
else process.env.STACKS_DIR = originalStacksDir;
await rm(stacksDir, { recursive: true, force: true });
});
const snapshot = (verdict: "ok" | "down" | "degraded"): HealthSnapshot => ({ verdict, reasons: verdict === "ok" ? [] : ["bad"], httpStatus: verdict === "down" ? null : 200, installed: true, installedAt: null, lastCronRunAt: null, queue: null });
function deps(overrides: Record<string, unknown> = {}) {
let verdict: "ok" | "down" | "degraded" = "down";
let clock = t0.getTime();
const sendEmail = vi.fn().mockResolvedValue({ ok: true, message: "sent" });
return {
sendEmail,
set: (v: typeof verdict) => (verdict = v),
advance: (min: number) => (clock += min * 60_000),
overrides: {
probe: async () => snapshot(verdict),
sendEmail,
now: () => new Date(clock),
isBusy: async () => false,
readSettings: async () => ({ ...DEFAULT_OPERATOR_SETTINGS, alertEmail: "ops@example.com" }),
describe: async () => ({ label: "Acme", domain: "acme.example.com" }),
...overrides,
},
};
}
it("emails once after two failures, once on recovery, and nothing in between", async () => {
const d = deps();
await runHealthCheck("acme", d.overrides);
expect(d.sendEmail).not.toHaveBeenCalled();
d.advance(3);
await runHealthCheck("acme", d.overrides);
expect(d.sendEmail).toHaveBeenCalledTimes(1);
expect(d.sendEmail.mock.calls[0]?.[0]).toContain("is DOWN");
d.advance(3);
await runHealthCheck("acme", d.overrides);
expect(d.sendEmail).toHaveBeenCalledTimes(1);
d.set("ok");
d.advance(3);
await runHealthCheck("acme", d.overrides);
expect(d.sendEmail).toHaveBeenCalledTimes(2);
expect(d.sendEmail.mock.calls[1]?.[0]).toContain("recovered");
});
it("repeats once a day while still bad", async () => {
const d = deps();
await runHealthCheck("acme", d.overrides);
d.advance(3);
await runHealthCheck("acme", d.overrides);
d.advance(23 * 60);
await runHealthCheck("acme", d.overrides);
expect(d.sendEmail).toHaveBeenCalledTimes(1);
d.advance(60 + 3);
await runHealthCheck("acme", d.overrides);
expect(d.sendEmail).toHaveBeenCalledTimes(2);
expect(d.sendEmail.mock.calls[1]?.[0]).toContain("Still DOWN");
});
it("retries an alert whose email failed, instead of losing it", async () => {
const d = deps();
d.sendEmail.mockResolvedValueOnce({ ok: false, message: "SMTP down" });
await runHealthCheck("acme", d.overrides);
d.advance(3);
await runHealthCheck("acme", d.overrides); // alert raised, send fails
expect(d.sendEmail).toHaveBeenCalledTimes(1);
d.advance(3);
await runHealthCheck("acme", d.overrides); // same state, but the unsent alert goes out now
expect(d.sendEmail).toHaveBeenCalledTimes(2);
d.advance(3);
await runHealthCheck("acme", d.overrides);
expect(d.sendEmail).toHaveBeenCalledTimes(2);
});
it("sends nothing when no alert address is set or the toggle is off", async () => {
const noAddress = deps({ readSettings: async () => ({ ...DEFAULT_OPERATOR_SETTINGS, alertEmail: "" }) });
await runHealthCheck("acme", noAddress.overrides);
noAddress.advance(3);
await runHealthCheck("acme", noAddress.overrides);
expect(noAddress.sendEmail).not.toHaveBeenCalled();
const toggledOff = deps({ readSettings: async () => ({ ...DEFAULT_OPERATOR_SETTINGS, alertEmail: "ops@example.com", alertOnHealth: false }) });
await runHealthCheck("zeta", toggledOff.overrides);
toggledOff.advance(3);
await runHealthCheck("zeta", toggledOff.overrides);
expect(toggledOff.sendEmail).not.toHaveBeenCalled();
});
it("doesn't probe a business that is mid-job or was stopped on purpose", async () => {
const probe = vi.fn().mockResolvedValue(snapshot("down"));
const busy = deps({ isBusy: async () => true, probe });
expect(await runHealthCheck("acme", busy.overrides)).toMatchObject({ skipped: expect.stringContaining("job") });
await setIntentionallyStopped("zeta", true);
const stopped = deps({ probe });
expect(await runHealthCheck("zeta", stopped.overrides)).toMatchObject({ skipped: "intentionally stopped" });
expect(probe).not.toHaveBeenCalled();
await setIntentionallyStopped("zeta", false);
await runHealthCheck("zeta", stopped.overrides);
expect(probe).toHaveBeenCalledTimes(1);
});
});
+141
View File
@@ -0,0 +1,141 @@
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { pruneOldJobs, readJob, recoverInterruptedJobs, startJob } from "../jobs";
import { BusinessLockedError, getActiveLock, readLock } from "../locks";
function deferred<T = void>() {
let resolve!: (value: T) => void;
const promise = new Promise<T>((r) => (resolve = r));
return { promise, resolve };
}
describe("job runner", () => {
let stacksDir: string;
const originalStacksDir = process.env.STACKS_DIR;
beforeEach(async () => {
stacksDir = await mkdtemp(path.join(tmpdir(), "nc-manager-jobs-"));
process.env.STACKS_DIR = stacksDir;
});
afterEach(async () => {
if (originalStacksDir === undefined) delete process.env.STACKS_DIR;
else process.env.STACKS_DIR = originalStacksDir;
await rm(stacksDir, { recursive: true, force: true });
});
const steps = [
{ id: "one", label: "First" },
{ id: "two", label: "Second" },
{ id: "three", label: "Third" },
];
it("writes a pollable record with step states and a log tail, holding the lock only while running", async () => {
const gate = deferred();
const { jobId, done } = await startJob({
kind: "update",
slug: "acme",
title: "Update acme",
steps,
run: async (ctx) => {
await ctx.step("one", async () => ctx.log("hello from step one"));
await ctx.step("two", async () => gate.promise);
ctx.skip("three", "not needed");
return { ok: true };
},
});
// Visible immediately, mid-run: step one done, step two running, lock held.
await new Promise((r) => setTimeout(r, 30));
const midway = await readJob(jobId);
expect(midway?.status).toBe("running");
expect(midway?.steps.map((s) => s.status)).toEqual(["done", "running", "pending"]);
expect(midway?.logTail.join("\n")).toContain("hello from step one");
expect((await getActiveLock("acme"))?.jobId).toBe(jobId);
gate.resolve();
const finished = await done;
expect(finished.status).toBe("succeeded");
const onDisk = await readJob(jobId);
expect(onDisk?.steps.map((s) => s.status)).toEqual(["done", "done", "skipped"]);
expect(onDisk?.result).toEqual({ ok: true });
expect(onDisk?.finishedAt).not.toBeNull();
expect(await readLock("acme")).toBeNull();
});
it("records a failure on the failing step, skips the rest, and still releases the lock", async () => {
const { jobId, done } = await startJob({
kind: "backup",
slug: "acme",
title: "Back up acme",
steps,
run: async (ctx) => {
await ctx.step("one", async () => undefined);
await ctx.step("two", async () => {
throw new Error("mysqldump exited with code 2");
});
},
});
const finished = await done;
expect(finished.status).toBe("failed");
expect(finished.error).toBe("mysqldump exited with code 2");
expect((await readJob(jobId))?.steps.map((s) => s.status)).toEqual(["done", "failed", "skipped"]);
expect((await readJob(jobId))?.steps[1]?.detail).toBe("mysqldump exited with code 2");
expect(await readLock("acme")).toBeNull();
});
it("a second job on the same business is rejected while the first runs, and allowed after", async () => {
const gate = deferred();
const first = await startJob({ kind: "backup", slug: "acme", title: "a", steps, run: async () => gate.promise });
await expect(startJob({ kind: "delete", slug: "acme", title: "b", steps, run: async () => undefined })).rejects.toBeInstanceOf(BusinessLockedError);
// A different business is unaffected.
const other = await startJob({ kind: "update", slug: "zeta", title: "c", steps, run: async () => undefined });
await other.done;
gate.resolve();
await first.done;
const again = await startJob({ kind: "delete", slug: "acme", title: "d", steps, run: async () => undefined });
expect((await again.done).status).toBe("succeeded");
});
it("rejects unknown job ids without touching the filesystem", async () => {
expect(await readJob("../../etc/passwd")).toBeNull();
expect(await readJob("nope")).toBeNull();
});
it("recoverInterruptedJobs fails jobs left 'running' by a dead process and frees their locks", async () => {
const jobsPath = path.join(stacksDir, "manager", "jobs");
const locksPath = path.join(stacksDir, "manager", "locks");
await mkdir(jobsPath, { recursive: true });
await mkdir(locksPath, { recursive: true });
const id = "abc123-deadbeef";
await writeFile(
path.join(jobsPath, `${id}.json`),
JSON.stringify({
id, kind: "restore", slug: "acme", title: "Restore acme", status: "running", createdAt: new Date().toISOString(), finishedAt: null,
steps: [{ id: "a", label: "A", status: "running", startedAt: null, finishedAt: null, detail: null }, { id: "b", label: "B", status: "pending", startedAt: null, finishedAt: null, detail: null }],
logTail: [], error: null, result: null,
}),
);
await writeFile(path.join(locksPath, "acme.json"), JSON.stringify({ slug: "acme", jobId: id, kind: "restore", acquiredAt: new Date().toISOString(), heartbeatAt: new Date().toISOString() }));
expect(await recoverInterruptedJobs()).toBe(1);
const job = await readJob(id);
expect(job?.status).toBe("failed");
expect(job?.error).toMatch(/Interrupted/);
expect(job?.steps.map((s) => s.status)).toEqual(["failed", "skipped"]);
expect(await readLock("acme")).toBeNull();
});
it("prunes only old, finished job records", async () => {
const { jobId, done } = await startJob({ kind: "update", slug: "acme", title: "x", steps, run: async () => undefined });
await done;
expect(await pruneOldJobs(14 * 24 * 3600 * 1000)).toBe(0);
expect(await pruneOldJobs(14 * 24 * 3600 * 1000, new Date(Date.now() + 15 * 24 * 3600 * 1000))).toBe(1);
expect(await readJob(jobId)).toBeNull();
expect(await readFile(path.join(stacksDir, "manager", "jobs", "x.json")).catch(() => null)).toBeNull();
});
});
+83
View File
@@ -0,0 +1,83 @@
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { acquireLock, assertNotLocked, BusinessLockedError, getActiveLock, isLockStale, readLock, releaseLock, STALE_LOCK_MS, touchLock } from "../locks";
describe("per-business lock", () => {
let stacksDir: string;
const originalStacksDir = process.env.STACKS_DIR;
beforeEach(async () => {
stacksDir = await mkdtemp(path.join(tmpdir(), "nc-manager-locks-"));
process.env.STACKS_DIR = stacksDir;
});
afterEach(async () => {
if (originalStacksDir === undefined) delete process.env.STACKS_DIR;
else process.env.STACKS_DIR = originalStacksDir;
await rm(stacksDir, { recursive: true, force: true });
});
it("lets one job hold a business and rejects a second one (Update vs Delete vs second backup)", async () => {
await acquireLock("acme", { jobId: "j1", kind: "backup" });
await expect(acquireLock("acme", { jobId: "j2", kind: "update" })).rejects.toBeInstanceOf(BusinessLockedError);
await expect(acquireLock("acme", { jobId: "j3", kind: "delete" })).rejects.toThrow(/backup job \(j1\)/);
await expect(assertNotLocked("acme")).rejects.toBeInstanceOf(BusinessLockedError);
});
it("locks are per business", async () => {
await acquireLock("acme", { jobId: "j1", kind: "backup" });
await expect(acquireLock("zeta", { jobId: "j2", kind: "backup" })).resolves.toMatchObject({ slug: "zeta" });
});
it("only two simultaneous acquirers can't both win", async () => {
const results = await Promise.allSettled([
acquireLock("acme", { jobId: "a", kind: "backup" }),
acquireLock("acme", { jobId: "b", kind: "update" }),
acquireLock("acme", { jobId: "c", kind: "restore" }),
]);
expect(results.filter((r) => r.status === "fulfilled")).toHaveLength(1);
});
it("releases only for the owner, then can be re-acquired", async () => {
await acquireLock("acme", { jobId: "j1", kind: "backup" });
await releaseLock("acme", "someone-else");
expect(await readLock("acme")).not.toBeNull();
await releaseLock("acme", "j1");
expect(await readLock("acme")).toBeNull();
await expect(acquireLock("acme", { jobId: "j2", kind: "update" })).resolves.toBeDefined();
});
describe("stale-lock rule", () => {
it("a lock is stale once its heartbeat is older than the threshold", () => {
const now = new Date("2026-10-17T12:00:00Z");
const lock = { slug: "acme", jobId: "j1", kind: "backup", acquiredAt: now.toISOString(), heartbeatAt: now.toISOString() };
expect(isLockStale(lock, new Date(now.getTime() + STALE_LOCK_MS - 1000))).toBe(false);
expect(isLockStale(lock, new Date(now.getTime() + STALE_LOCK_MS + 1000))).toBe(true);
expect(isLockStale({ ...lock, heartbeatAt: "garbage" }, now)).toBe(true);
});
it("a stale lock is taken over by the next job; a fresh one is not", async () => {
const old = new Date(Date.now() - STALE_LOCK_MS - 60_000);
await acquireLock("acme", { jobId: "dead", kind: "restore" }, old);
expect(await getActiveLock("acme")).toBeNull();
await expect(assertNotLocked("acme")).resolves.toBeUndefined();
const taken = await acquireLock("acme", { jobId: "alive", kind: "update" });
expect(taken.jobId).toBe("alive");
await expect(acquireLock("acme", { jobId: "third", kind: "backup" })).rejects.toBeInstanceOf(BusinessLockedError);
});
it("a heartbeat keeps a long job's lock alive; a dead owner can't release its successor's lock", async () => {
const start = new Date(Date.now() - STALE_LOCK_MS - 60_000);
await acquireLock("acme", { jobId: "slow", kind: "backup" }, start);
await touchLock("acme", "slow", new Date());
expect(await getActiveLock("acme")).not.toBeNull();
// Pretend it went stale and was taken over.
await releaseLock("acme", "slow");
await acquireLock("acme", { jobId: "successor", kind: "update" });
await releaseLock("acme", "slow");
expect((await readLock("acme"))?.jobId).toBe("successor");
});
});
});
@@ -0,0 +1,105 @@
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
const calls: string[] = [];
vi.mock("@/lib/docker", () => ({
composeUp: vi.fn(async (_dir: string, opts?: unknown) => calls.push(`composeUp:${JSON.stringify(opts)}`)),
}));
const waitForHealthMock = vi.fn();
vi.mock("@/lib/health", () => ({ waitForHealth: (...a: unknown[]) => waitForHealthMock(...a) }));
const performBackupMock = vi.fn();
vi.mock("@/lib/backup", () => ({ performBackup: (...a: unknown[]) => performBackupMock(...a), BACKUP_STEPS: [] }));
const readBackupPassphraseMock = vi.fn();
vi.mock("@/lib/global-settings", () => ({
readBackupPassphrase: () => readBackupPassphraseMock(),
readOperatorSettings: vi.fn(async () => ({ alertEmail: "", alertOnBackupFailure: true })),
}));
vi.mock("@/lib/provisioning", () => ({ deleteBusiness: vi.fn(async () => (calls.push("deleteBusiness"), { slug: "acme", warnings: [] })) }));
vi.mock("@/lib/operator-email", () => ({ sendOperatorEmail: vi.fn() }));
import { BusinessLockedError } from "../locks";
import { startDeleteJob, startUpdateJob } from "../operations";
import { startJob } from "../jobs";
import { readJob } from "../jobs";
describe("Update and Delete jobs", () => {
let stacksDir: string;
const originalStacksDir = process.env.STACKS_DIR;
beforeEach(async () => {
stacksDir = await mkdtemp(path.join(tmpdir(), "nc-manager-operations-"));
process.env.STACKS_DIR = stacksDir;
await mkdir(path.join(stacksDir, "acme"), { recursive: true });
await writeFile(path.join(stacksDir, "acme", "compose.yaml"), "name: x\n");
calls.length = 0;
waitForHealthMock.mockReset().mockImplementation(async () => (calls.push("waitForHealth"), undefined));
performBackupMock.mockReset().mockImplementation(async (_slug: string, kind: string) => (calls.push(`backup:${kind}`), { fileName: `f-${kind}.7z` }));
readBackupPassphraseMock.mockReset().mockResolvedValue("a-stored-passphrase");
});
afterEach(async () => {
if (originalStacksDir === undefined) delete process.env.STACKS_DIR;
else process.env.STACKS_DIR = originalStacksDir;
await rm(stacksDir, { recursive: true, force: true });
});
it("Update without backup: recreate container → wait for /api/health → done", async () => {
const { jobId, done } = await startUpdateJob("acme", false);
const job = await done;
expect(job.status).toBe("succeeded");
expect(job.steps.map((s) => [s.id, s.status])).toEqual([["recreate", "done"], ["health", "done"], ["done", "done"]]);
expect(calls).toEqual(['composeUp:{"forceRecreate":true}', "waitForHealth"]);
expect((await readJob(jobId))?.title).toBe("Update acme");
});
it("Update with 'back up first': a safety backup runs before the container is touched", async () => {
const { done } = await startUpdateJob("acme", true);
const job = await done;
expect(job.steps.map((s) => s.id)).toEqual(["backup", "recreate", "health", "done"]);
expect(calls).toEqual(["backup:safety", 'composeUp:{"forceRecreate":true}', "waitForHealth"]);
});
it("a failed pre-update backup stops the update before the container is recreated", async () => {
performBackupMock.mockRejectedValue(new Error("disk full"));
const job = await (await startUpdateJob("acme", true)).done;
expect(job.status).toBe("failed");
expect(job.error).toBe("disk full");
expect(calls.some((c) => c.startsWith("composeUp"))).toBe(false);
expect(job.steps.map((s) => s.status)).toEqual(["failed", "skipped", "skipped", "skipped"]);
});
it("an unhealthy app after the recreate fails the job with the health error", async () => {
waitForHealthMock.mockRejectedValue(new Error('"acme" did not become healthy within 180s.'));
const job = await (await startUpdateJob("acme", false)).done;
expect(job.status).toBe("failed");
expect(job.steps.map((s) => s.status)).toEqual(["done", "failed", "skipped"]);
});
it("Update refuses to run on top of another job for the same business", async () => {
let release!: () => void;
const gate = new Promise<void>((r) => (release = r));
const running = await startJob({ kind: "backup", slug: "acme", title: "x", steps: [], run: async () => gate });
await expect(startUpdateJob("acme", false)).rejects.toBeInstanceOf(BusinessLockedError);
await expect(startDeleteJob("acme", false)).rejects.toBeInstanceOf(BusinessLockedError);
release();
await running.done;
});
it("Delete with a final backup: backs up first, deletes only afterwards; a failed backup aborts the delete", async () => {
const ok = await (await startDeleteJob("acme", true)).done;
expect(ok.status).toBe("succeeded");
expect(calls).toEqual(["backup:final", "deleteBusiness"]);
calls.length = 0;
performBackupMock.mockRejectedValue(new Error("mysqldump failed"));
const failed = await (await startDeleteJob("acme", true)).done;
expect(failed.status).toBe("failed");
expect(calls).not.toContain("deleteBusiness");
});
it("Delete without a final backup skips that step", async () => {
const job = await (await startDeleteJob("acme", false)).done;
expect(job.steps.map((s) => s.status)).toEqual(["skipped", "done"]);
expect(calls).toEqual(["deleteBusiness"]);
});
});
+71
View File
@@ -0,0 +1,71 @@
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { CommandError, runCapture, runFromFile, runPipeline, runToFile } from "../process";
const node = (script: string, env?: Record<string, string>) => ({ command: process.execPath, args: ["-e", script], env });
describe("streaming subprocess helpers", () => {
let dir: string;
beforeEach(async () => {
dir = await mkdtemp(path.join(tmpdir(), "nc-manager-process-"));
});
afterEach(async () => {
await rm(dir, { recursive: true, force: true });
});
it("runToFile streams far more than execFile's default buffer without buffering it", async () => {
const out = path.join(dir, "big.bin");
// ~30 MB, written in chunks — would blow execFile's 1 MB default maxBuffer.
await runToFile(node("const b=Buffer.alloc(1024*1024,97);let n=30;(function w(){while(n>0){n--;if(!process.stdout.write(b))return process.stdout.once('drain',w)}})()"), out);
expect((await readFile(out)).length).toBe(30 * 1024 * 1024);
});
it("runToFile rejects on a non-zero exit with the stderr tail (and no command arguments)", async () => {
const out = path.join(dir, "x");
const error = await runToFile(node("console.error('boom: access denied');process.exit(3)", { MYSQL_PWD: "hunter2" }), out).catch((e: unknown) => e);
expect(error).toBeInstanceOf(CommandError);
expect((error as CommandError).exitCode).toBe(3);
expect((error as CommandError).message).toContain("boom: access denied");
expect((error as CommandError).message).not.toContain("hunter2");
});
it("passes env to the child but not via arguments", async () => {
const { stdout } = await runCapture(node("process.stdout.write(process.env.MYSQL_PWD)", { MYSQL_PWD: "from-env" }));
expect(stdout).toBe("from-env");
});
it("runFromFile feeds a file to stdin", async () => {
const input = path.join(dir, "in.txt");
const output = path.join(dir, "out.txt");
await writeFile(input, "line one\nline two\n");
await runFromFile(node(`require('fs').writeFileSync(${JSON.stringify(output)}, require('fs').readFileSync(0))`), input);
expect(await readFile(output, "utf8")).toBe("line one\nline two\n");
});
it("runPipeline pipes the producer's stdout into the consumer's stdin", async () => {
const output = path.join(dir, "piped.txt");
await runPipeline(
node("process.stdout.write('hello pipeline')"),
node(`require('fs').writeFileSync(${JSON.stringify(output)}, require('fs').readFileSync(0))`),
);
expect(await readFile(output, "utf8")).toBe("hello pipeline");
});
it("runPipeline fails when the producer fails even if the consumer 'succeeds' on the truncated stream", async () => {
await expect(runPipeline(node("process.stdout.write('partial');console.error('Wrong password');process.exit(2)"), node("require('fs').readFileSync(0)"))).rejects.toThrow(/Wrong password/);
});
it("runPipeline fails when the consumer fails", async () => {
await expect(runPipeline(node("process.stdout.write('x'.repeat(1000))"), node("console.error('ERROR 1064 syntax');process.exit(1)"))).rejects.toThrow(/1064/);
});
it("reports a missing binary clearly", async () => {
await expect(runCapture({ command: "definitely-not-a-real-binary-xyz", args: [] })).rejects.toThrow(/Couldn't run definitely-not-a-real-binary-xyz/);
});
it("runCapture caps output", async () => {
await expect(runCapture(node("process.stdout.write('x'.repeat(5000))"), { maxBytes: 100 })).rejects.toThrow(/more than 100 bytes/);
});
});
+225
View File
@@ -0,0 +1,225 @@
import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { CommandSpec } from "../process";
/** Every side effect, in the order it happens — the restore's contract. */
const calls: string[] = [];
const record = (label: string) => calls.push(label);
vi.mock("@/lib/docker", () => ({
composeDown: vi.fn(async (_dir: string, opts?: unknown) => record(`composeDown${opts ? ":" + JSON.stringify(opts) : ""}`)),
composeCreate: vi.fn(async () => record("composeCreate")),
composeUp: vi.fn(async () => record("composeUp")),
listImageMigrations: vi.fn(async () => {
record("listImageMigrations");
return ["20260912165903_add_staff_selected_by_customer", "20261017180000_add_business_holidays"];
}),
}));
const assertDomainAvailableMock = vi.fn();
const recreateDatabaseMock = vi.fn();
vi.mock("@/lib/provisioning", () => ({
assertDomainAvailable: (...a: unknown[]) => assertDomainAvailableMock(...a),
recreateDatabaseAndUser: (...a: unknown[]) => recreateDatabaseMock(...a),
writeCaddySiteAndReload: vi.fn(async (slug: string, domain: string) => record(`writeCaddy:${slug}:${domain}`)),
}));
const runPipelineMock = vi.fn();
vi.mock("@/lib/process", () => ({ runPipeline: (...a: unknown[]) => runPipelineMock(...a) }));
const sevenZipTestMock = vi.fn();
const sevenZipReadMemberMock = vi.fn();
vi.mock("@/lib/sevenzip", async (importOriginal) => ({
...(await importOriginal<typeof import("../sevenzip")>()),
sevenZipTest: (...a: unknown[]) => sevenZipTestMock(...a),
sevenZipReadMember: (...a: unknown[]) => sevenZipReadMemberMock(...a),
}));
const waitForHealthMock = vi.fn();
vi.mock("@/lib/health", () => ({ waitForHealth: (...a: unknown[]) => waitForHealthMock(...a) }));
const performBackupMock = vi.fn();
vi.mock("@/lib/backup", () => ({ performBackup: (...a: unknown[]) => performBackupMock(...a) }));
import { performRestore, RESTORE_STEPS } from "../restore";
const PASSPHRASE = "restore-passphrase-1";
const DB_PASSWORD = "f".repeat(32);
const ARCHIVE = "/stacks/backups/acme/acme__20261017T030000Z__regular.7z";
const BACKED_UP_ENV = ['BUSINESS_NAME="Acme Barbers"', 'DOMAIN="acme.example.com"', 'DATABASE_NAME="acme"', 'DATABASE_USER="acme"', `DATABASE_PASSWORD="${DB_PASSWORD}"`, 'AUTH_SECRET="s3cret-from-backup"'].join("\n") + "\n";
function manifest(over: Record<string, unknown> = {}) {
return JSON.stringify({
formatVersion: 1, slug: "acme", name: "Acme Barbers", domain: "acme.example.com", databaseName: "acme", databaseUser: "acme",
createdAt: "2026-10-17T03:00:00.000Z", kind: "regular", lastMigration: "20261017180000_add_business_holidays", appImage: null, hasCaddySite: true, ...over,
});
}
describe("performRestore", () => {
let stacksDir: string;
const originalStacksDir = process.env.STACKS_DIR;
const originalRoot = process.env.MYSQL_ROOT_PASSWORD;
beforeEach(async () => {
stacksDir = await mkdtemp(path.join(tmpdir(), "nc-manager-restore-"));
process.env.STACKS_DIR = stacksDir;
process.env.MYSQL_ROOT_PASSWORD = "root-pw";
calls.length = 0;
assertDomainAvailableMock.mockReset().mockImplementation(async () => record("assertDomainAvailable"));
recreateDatabaseMock.mockReset().mockImplementation(async () => record("recreateDatabase"));
runPipelineMock.mockReset().mockImplementation(async (source: CommandSpec) => record(`pipeline:${source.args.at(-1)}`));
sevenZipTestMock.mockReset().mockImplementation(async () => record("7zz-test"));
sevenZipReadMemberMock.mockReset().mockImplementation(async (_a: string, _p: string, member: string) => {
record(`read:${member}`);
return member === "manifest.json" ? manifest() : BACKED_UP_ENV;
});
waitForHealthMock.mockReset().mockImplementation(async (_slug: string, opts: unknown) => record(`waitForHealth:${JSON.stringify((opts as { requireInstalled?: boolean }).requireInstalled)}`));
performBackupMock.mockReset().mockImplementation(async (slug: string, kind: string) => {
record(`safetyBackup:${slug}:${kind}`);
return { fileName: "acme__20261017T040000Z__safety.7z" };
});
});
afterEach(async () => {
if (originalStacksDir === undefined) delete process.env.STACKS_DIR;
else process.env.STACKS_DIR = originalStacksDir;
if (originalRoot === undefined) delete process.env.MYSQL_ROOT_PASSWORD;
else process.env.MYSQL_ROOT_PASSWORD = originalRoot;
await rm(stacksDir, { recursive: true, force: true });
});
function makeCtx() {
const steps: string[] = [];
const lines: string[] = [];
return {
steps, lines,
ctx: {
log: (l: string) => lines.push(l),
step: async <T,>(id: string, fn: () => Promise<T>) => (steps.push(`step:${id}`), fn()),
skip: (id: string) => steps.push(`skip:${id}`),
},
};
}
async function existingStack() {
await mkdir(path.join(stacksDir, "acme"), { recursive: true });
await writeFile(path.join(stacksDir, "acme", "compose.yaml"), "name: old\n");
await writeFile(path.join(stacksDir, "acme", ".env"), 'DOMAIN="acme.example.com"\nDATABASE_NAME="acme"\n');
}
it("fresh server: runs the steps in the planned order and the side effects in the planned order", async () => {
const { ctx, steps } = makeCtx();
const result = await performRestore({ archivePath: ARCHIVE, passphrase: PASSPHRASE, replace: false, ctx });
expect(steps).toEqual([
"step:verify", "step:check", "skip:safety", "skip:stop", "step:database", "step:import", "step:config", "step:media", "step:start", "step:health",
]);
// The declared step list (what the UI renders) matches what actually runs.
expect(RESTORE_STEPS.map((s) => `${s.id}`)).toEqual(["verify", "check", "safety", "stop", "database", "import", "config", "media", "start", "health"]);
expect(calls).toEqual([
"7zz-test", "read:manifest.json", "read:config/.env",
"assertDomainAvailable", "listImageMigrations",
"recreateDatabase",
"pipeline:db.sql",
"writeCaddy:acme:acme.example.com",
"composeCreate", "pipeline:media.tar",
"composeUp",
"waitForHealth:true",
]);
expect(result).toEqual({ slug: "acme", domain: "acme.example.com", safetyBackup: null });
});
it("recreates the DB with the password FROM THE BACKED-UP .env, and writes that .env verbatim (0600) plus a regenerated compose.yaml", async () => {
await performRestore({ archivePath: ARCHIVE, passphrase: PASSPHRASE, replace: false, ctx: makeCtx().ctx });
expect(recreateDatabaseMock).toHaveBeenCalledWith("acme", "acme", DB_PASSWORD);
const envPath = path.join(stacksDir, "acme", ".env");
expect(await readFile(envPath, "utf8")).toBe(BACKED_UP_ENV);
expect((await stat(envPath)).mode & 0o777).toBe(0o600);
const compose = await readFile(path.join(stacksDir, "acme", "compose.yaml"), "utf8");
expect(compose).toContain("container_name: next-chair-acme");
expect(compose).toContain("next-chair-media-acme:/app/storage/media");
});
it("builds the import and media pipelines as argument arrays: 7zz -so → docker, no password in argv", async () => {
await performRestore({ archivePath: ARCHIVE, passphrase: PASSPHRASE, replace: false, ctx: makeCtx().ctx });
const [dumpSource, dumpSink] = runPipelineMock.mock.calls[0] as [CommandSpec, CommandSpec];
expect(dumpSource.command).toBe("7zz");
expect(dumpSource.args).toEqual(["x", "-so", "-t7z", `-p${PASSPHRASE}`, "-y", "-bd", "-bso0", "-bsp0", ARCHIVE, "db.sql"]);
expect(dumpSink).toMatchObject({ command: "docker" });
expect(dumpSink.args).toEqual(["exec", "-i", "-e", "MYSQL_PWD", "next-chair-mysql", "mysql", "-uroot", "--default-character-set=utf8mb4", "acme"]);
expect(dumpSink.args.join(" ")).not.toContain("root-pw");
const [mediaSource, mediaSink] = runPipelineMock.mock.calls[1] as [CommandSpec, CommandSpec];
expect(mediaSource.args.at(-1)).toBe("media.tar");
expect(mediaSink.args).toContain("--user");
expect(mediaSink.args.at(-1)).toContain("chown -R 1001:1001");
});
it("existing business + replace: safety backup FIRST, then compose down (with volumes), then everything else", async () => {
await existingStack();
const { ctx, steps } = makeCtx();
const result = await performRestore({ archivePath: ARCHIVE, passphrase: PASSPHRASE, replace: true, ctx });
expect(steps).toEqual([
"step:verify", "step:check", "step:safety", "step:stop", "step:database", "step:import", "step:config", "step:media", "step:start", "step:health",
]);
const safety = calls.indexOf("safetyBackup:acme:safety");
const down = calls.findIndex((c) => c.startsWith("composeDown"));
expect(safety).toBeGreaterThan(-1);
expect(down).toBeGreaterThan(safety);
expect(calls[down]).toBe('composeDown:{"removeVolumes":true}');
expect(calls.indexOf("recreateDatabase")).toBeGreaterThan(down);
expect(result.safetyBackup).toBe("acme__20261017T040000Z__safety.7z");
});
it("existing business without 'replace': refuses before touching anything", async () => {
await existingStack();
await expect(performRestore({ archivePath: ARCHIVE, passphrase: PASSPHRASE, replace: false, ctx: makeCtx().ctx })).rejects.toThrow(/already exists.*replace/);
expect(performBackupMock).not.toHaveBeenCalled();
expect(recreateDatabaseMock).not.toHaveBeenCalled();
expect(calls.some((c) => c.startsWith("composeDown"))).toBe(false);
});
it("domain conflict: refuses before any destructive step", async () => {
assertDomainAvailableMock.mockRejectedValue(new Error('Domain "acme.example.com" is already in use by "other".'));
await expect(performRestore({ archivePath: ARCHIVE, passphrase: PASSPHRASE, replace: false, ctx: makeCtx().ctx })).rejects.toThrow(/already in use/);
expect(recreateDatabaseMock).not.toHaveBeenCalled();
expect(runPipelineMock).not.toHaveBeenCalled();
});
it("version guard: a backup from a newer schema than the image is refused before any change", async () => {
sevenZipReadMemberMock.mockImplementation(async (_a: string, _p: string, member: string) =>
member === "manifest.json" ? manifest({ lastMigration: "20271231000000_from_the_future" }) : BACKED_UP_ENV,
);
await expect(performRestore({ archivePath: ARCHIVE, passphrase: PASSPHRASE, replace: false, ctx: makeCtx().ctx })).rejects.toThrow(/newer than the running app image/);
expect(recreateDatabaseMock).not.toHaveBeenCalled();
});
it("refuses an archive whose .env disagrees with its manifest", async () => {
sevenZipReadMemberMock.mockImplementation(async (_a: string, _p: string, member: string) =>
member === "manifest.json" ? manifest() : BACKED_UP_ENV.replace('DATABASE_NAME="acme"', 'DATABASE_NAME="other"'),
);
await expect(performRestore({ archivePath: ARCHIVE, passphrase: PASSPHRASE, replace: false, ctx: makeCtx().ctx })).rejects.toThrow(/doesn't match its manifest/);
expect(recreateDatabaseMock).not.toHaveBeenCalled();
});
it("a bad archive or wrong passphrase fails in the first step", async () => {
sevenZipTestMock.mockRejectedValue(new Error("7zz exited with code 2: Wrong password"));
await expect(performRestore({ archivePath: ARCHIVE, passphrase: PASSPHRASE, replace: false, ctx: makeCtx().ctx })).rejects.toThrow(/Wrong password/);
expect(assertDomainAvailableMock).not.toHaveBeenCalled();
});
it("never logs the passphrase or any .env secret", async () => {
const { ctx, lines } = makeCtx();
await performRestore({ archivePath: ARCHIVE, passphrase: PASSPHRASE, replace: false, ctx });
const all = lines.join("\n");
expect(all).not.toContain(PASSPHRASE);
expect(all).not.toContain(DB_PASSWORD);
expect(all).not.toContain("s3cret-from-backup");
});
});
+161
View File
@@ -0,0 +1,161 @@
import { describe, expect, it, vi } from "vitest";
import { MemoryStateStore, runTick, taskKey, type SchedulerConfig, type SchedulerTask } from "../scheduler";
const config: SchedulerConfig = { notificationsIntervalSeconds: 30, healthIntervalMinutes: 3 };
function harness(tasks: SchedulerTask[], slugs: string[] = ["acme", "zeta"]) {
let nowMs = Date.parse("2026-10-17T12:00:00Z");
const store = new MemoryStateStore();
const inFlight = new Set<string>();
const tick = () =>
runTick({ tasks, store, listSlugs: async () => slugs, config: async () => config, now: () => new Date(nowMs), inFlight });
return { store, tick, advance: (ms: number) => (nowMs += ms), inFlight };
}
function task(overrides: Partial<SchedulerTask> & { run?: SchedulerTask["run"] } = {}): SchedulerTask & { calls: (string | null)[] } {
const calls: (string | null)[] = [];
return {
id: "notifications",
label: "n",
scope: "business",
intervalMs: (c) => c.notificationsIntervalSeconds * 1000,
run: async ({ slug }) => {
calls.push(slug);
return { ok: true, message: "HTTP 200" };
},
...overrides,
calls,
} as SchedulerTask & { calls: (string | null)[] };
}
describe("scheduler due-time", () => {
it("runs a per-business task for every business on the first tick", async () => {
const t = task();
const h = harness([t]);
const { started, done } = await h.tick();
await done;
expect(started.sort()).toEqual(["notifications:acme", "notifications:zeta"]);
expect(t.calls.sort()).toEqual(["acme", "zeta"]);
});
it("does not run again until the task's interval has elapsed (30s default for notifications)", async () => {
const t = task();
const h = harness([t], ["acme"]);
await (await h.tick()).done;
h.advance(29_000);
await (await h.tick()).done;
expect(t.calls).toHaveLength(1);
h.advance(1_000);
await (await h.tick()).done;
expect(t.calls).toHaveLength(2);
});
it("uses the configured interval, per task", async () => {
const fast = task({ id: "fast", intervalMs: () => 10_000 });
const slow = task({ id: "slow", intervalMs: () => 60_000 });
const h = harness([fast, slow], ["acme"]);
await (await h.tick()).done;
h.advance(10_000);
await (await h.tick()).done;
expect(fast.calls).toHaveLength(2);
expect(slow.calls).toHaveLength(1);
});
it("a once-a-day task survives a restart: persisted lastStartedAt is honoured", async () => {
const daily = task({ id: "reviews", intervalMs: () => 24 * 3600 * 1000 });
const h = harness([daily], ["acme"]);
h.store.patch(taskKey("reviews", "acme"), { lastStartedAt: new Date(Date.parse("2026-10-17T12:00:00Z") - 3600_000).toISOString() });
await (await h.tick()).done;
expect(daily.calls).toHaveLength(0);
h.advance(23 * 3600 * 1000);
await (await h.tick()).done;
expect(daily.calls).toHaveLength(1);
});
it("a custom isDue (scheduled backups) overrides the interval", async () => {
const isDue = vi.fn().mockResolvedValueOnce(false).mockResolvedValueOnce(true);
const backups = task({ id: "backups", isDue });
const h = harness([backups], ["acme"]);
await (await h.tick()).done;
expect(backups.calls).toHaveLength(0);
h.advance(5_000);
await (await h.tick()).done;
expect(backups.calls).toEqual(["acme"]);
});
it("global tasks run once, not once per business", async () => {
const g = task({ id: "maintenance", scope: "global", intervalMs: () => 3600_000 });
const h = harness([g]);
await (await h.tick()).done;
expect(g.calls).toEqual([null]);
});
});
describe("scheduler overlap guard and results", () => {
it("never starts a task/business pair that is still in flight, however many ticks pass", async () => {
let release!: () => void;
const gate = new Promise<void>((r) => (release = r));
const slow = task({
run: async ({ slug }) => {
(slow as unknown as { calls: (string | null)[] }).calls.push(slug);
await gate;
return { ok: true, message: "done" };
},
});
const h = harness([slow], ["acme", "zeta"]);
const first = await h.tick();
expect(first.started).toHaveLength(2);
// Plenty of time passes while both are still running...
h.advance(10 * 60_000);
const second = await h.tick();
expect(second.started).toHaveLength(0);
expect(h.inFlight.size).toBe(2);
release();
await first.done;
expect(h.inFlight.size).toBe(0);
h.advance(60_000);
expect((await h.tick()).started).toHaveLength(2);
});
it("one slow business doesn't block another: guards are per business", async () => {
let release!: () => void;
const gate = new Promise<void>((r) => (release = r));
const calls: string[] = [];
const t = task({
run: async ({ slug }) => {
calls.push(slug!);
if (slug === "acme") await gate;
return { ok: true, message: "ok" };
},
});
const h = harness([t], ["acme", "zeta"]);
await h.tick();
h.advance(30_000);
const second = await h.tick();
expect(second.started).toEqual(["notifications:zeta"]);
release();
await second.done;
});
it("records the last result, including a thrown error, and keeps running other tasks", async () => {
const boom = task({ id: "boom", run: async () => { throw new Error("kaput"); } });
const fine = task({ id: "fine" });
const h = harness([boom, fine], ["acme"]);
await (await h.tick()).done;
const tasks = h.store.get().tasks;
expect(tasks["boom:acme"]).toMatchObject({ lastOk: false, lastMessage: "kaput" });
expect(tasks["fine:acme"]).toMatchObject({ lastOk: true, lastMessage: "HTTP 200" });
expect(h.store.get().heartbeatAt).not.toBeNull();
});
it("forgets state for businesses that no longer exist", async () => {
const t = task();
const h = harness([t], ["acme"]);
h.store.patch("notifications:gone", { lastStartedAt: "2026-01-01T00:00:00Z" });
await (await h.tick()).done;
expect(h.store.get().tasks["notifications:gone"]).toBeUndefined();
expect(h.store.get().tasks["notifications:acme"]).toBeDefined();
});
});
+147
View File
@@ -0,0 +1,147 @@
import { isValidSlug } from "./slug";
/**
* The manifest.json at the root of every backup archive — what a restore
* (possibly on a brand-new server that has never seen this business)
* learns about the backup before touching anything.
*/
export const BACKUP_FORMAT_VERSION = 1;
export const BACKUP_MEMBERS = {
manifest: "manifest.json",
database: "db.sql",
media: "media.tar",
env: "config/.env",
compose: "config/compose.yaml",
caddy: "config/site.caddy",
} as const;
export type BackupKind = "regular" | "safety" | "final";
export const BACKUP_KINDS: readonly BackupKind[] = ["regular", "safety", "final"];
export interface BackupManifest {
formatVersion: number;
slug: string;
name: string | null;
domain: string;
databaseName: string;
databaseUser: string;
createdAt: string;
kind: BackupKind;
/** Newest applied Prisma migration in the dumped database; null if it couldn't be read. */
lastMigration: string | null;
appImage: string | null;
hasCaddySite: boolean;
}
export class ManifestError extends Error {
constructor(message: string) {
super(message);
this.name = "ManifestError";
}
}
const SAFE_DB_IDENTIFIER = /^[a-z][a-z0-9_]{1,48}[a-z0-9]$/;
const MIGRATION_NAME = /^\d{14}_[A-Za-z0-9_]+$/;
function requireString(obj: Record<string, unknown>, key: string): string {
const value = obj[key];
if (typeof value !== "string" || !value.trim()) throw new ManifestError(`manifest.json is missing "${key}".`);
return value;
}
export function buildManifest(input: Omit<BackupManifest, "formatVersion">): BackupManifest {
return { formatVersion: BACKUP_FORMAT_VERSION, ...input };
}
/** Strict parse — a restore trusts nothing from the archive: slug, DB names and domain all end up in file paths or SQL. */
export function parseManifest(raw: string): BackupManifest {
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
throw new ManifestError("manifest.json isn't valid JSON — this doesn't look like a NextChair backup.");
}
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) throw new ManifestError("manifest.json has an unexpected shape.");
const obj = parsed as Record<string, unknown>;
const formatVersion = obj.formatVersion;
if (typeof formatVersion !== "number" || !Number.isInteger(formatVersion) || formatVersion < 1) {
throw new ManifestError("manifest.json has no valid formatVersion.");
}
if (formatVersion > BACKUP_FORMAT_VERSION) {
throw new ManifestError(
`This backup uses format v${formatVersion}, newer than this Manager understands (v${BACKUP_FORMAT_VERSION}). Update the Manager first.`,
);
}
const slug = requireString(obj, "slug");
if (!isValidSlug(slug)) throw new ManifestError(`The backup's slug "${slug}" isn't valid.`);
const domain = requireString(obj, "domain").trim().toLowerCase();
if (/["\s]/.test(domain)) throw new ManifestError("The backup's domain isn't valid.");
const databaseName = requireString(obj, "databaseName");
const databaseUser = requireString(obj, "databaseUser");
if (!SAFE_DB_IDENTIFIER.test(databaseName) || !SAFE_DB_IDENTIFIER.test(databaseUser)) {
throw new ManifestError("The backup names an unsafe database or user identifier.");
}
const createdAt = requireString(obj, "createdAt");
if (Number.isNaN(Date.parse(createdAt))) throw new ManifestError("The backup's createdAt isn't a valid date.");
const kind = obj.kind;
if (kind !== "regular" && kind !== "safety" && kind !== "final") throw new ManifestError("The backup has an unknown kind.");
const lastMigration = obj.lastMigration;
if (lastMigration !== null && lastMigration !== undefined) {
if (typeof lastMigration !== "string" || !MIGRATION_NAME.test(lastMigration)) throw new ManifestError("The backup's lastMigration isn't valid.");
}
const name = typeof obj.name === "string" && obj.name.trim() ? obj.name : null;
if (name && (name.includes('"') || name.includes("\n") || name.includes("\r"))) throw new ManifestError("The backup's business name isn't valid.");
return {
formatVersion,
slug,
name,
domain,
databaseName,
databaseUser,
createdAt,
kind,
lastMigration: typeof lastMigration === "string" ? lastMigration : null,
appImage: typeof obj.appImage === "string" ? obj.appImage : null,
hasCaddySite: obj.hasCaddySite === true,
};
}
/** Migration folders start with a 14-digit timestamp, so the prefix orders them. */
function migrationStamp(name: string): string {
return name.slice(0, 14);
}
/**
* Version guard: a dump taken on a newer schema than the running image
* knows would boot an app that can't read its own database. Refuse when
* the backup's last migration is newer than the image's newest. (An older
* dump is fine — the container's entrypoint runs `prisma migrate deploy`.)
*/
export function checkMigrationCompatibility(
backupLastMigration: string | null,
imageMigrations: string[],
): { ok: true } | { ok: false; reason: string } {
if (!backupLastMigration) return { ok: true };
if (imageMigrations.length === 0) {
return { ok: false, reason: "Couldn't read the migration list from the app image, so the schema version can't be checked. Is the app image built?" };
}
const newestInImage = [...imageMigrations].sort().at(-1)!;
if (migrationStamp(backupLastMigration) > migrationStamp(newestInImage)) {
return {
ok: false,
reason: `This backup's database is on migration "${backupLastMigration}", newer than the running app image's newest ("${newestInImage}"). Rebuild/update the app image first, then restore.`,
};
}
return { ok: true };
}
+173
View File
@@ -0,0 +1,173 @@
import { DEFAULT_KEEP, MAX_KEEP } from "./backup-store";
import { readJsonFile, writeJsonFileAtomic } from "./json-file";
import { backupSettingsPath } from "./manager-paths";
import { withMutex } from "./singleton";
/**
* Per-business backup schedule, in one manager-side file keyed by slug
* (STACKS_DIR/manager/backup-settings.json) — never inside a business's
* own stack directory, so it survives that stack being deleted and
* restored.
*
* Times are the Manager container's LOCAL time (set TZ on the container
* to change it; the default is UTC).
*/
export type BackupFrequency = "daily" | "weekly";
export interface BackupSettings {
enabled: boolean;
frequency: BackupFrequency;
/** "HH:MM", 24h, Manager-local. */
time: string;
/** 0 = Sunday … 6 = Saturday; only used when weekly. */
weekday: number;
/** How many regular backups to keep (newest first). */
keep: number;
enabledAt: string | null;
lastAttemptAt: string | null;
lastSuccessAt: string | null;
lastStatus: "ok" | "failed" | null;
lastMessage: string | null;
}
export const DEFAULT_BACKUP_SETTINGS: BackupSettings = {
enabled: false,
frequency: "daily",
time: "03:00",
weekday: 0,
keep: DEFAULT_KEEP,
enabledAt: null,
lastAttemptAt: null,
lastSuccessAt: null,
lastStatus: null,
lastMessage: null,
};
interface BackupSettingsFile {
businesses: Record<string, Partial<BackupSettings>>;
}
const TIME_PATTERN = /^([01]\d|2[0-3]):([0-5]\d)$/;
export function isValidBackupTime(value: string): boolean {
return TIME_PATTERN.test(value);
}
function normalize(raw: Partial<BackupSettings> | undefined): BackupSettings {
const merged = { ...DEFAULT_BACKUP_SETTINGS, ...(raw ?? {}) };
return {
...merged,
frequency: merged.frequency === "weekly" ? "weekly" : "daily",
time: isValidBackupTime(merged.time) ? merged.time : DEFAULT_BACKUP_SETTINGS.time,
weekday: Number.isInteger(merged.weekday) && merged.weekday >= 0 && merged.weekday <= 6 ? merged.weekday : 0,
keep: Number.isInteger(merged.keep) ? Math.min(MAX_KEEP, Math.max(1, merged.keep)) : DEFAULT_KEEP,
};
}
async function readFile(): Promise<BackupSettingsFile> {
const file = await readJsonFile<BackupSettingsFile>(backupSettingsPath(), { businesses: {} });
return { businesses: file.businesses ?? {} };
}
export async function readBackupSettings(slug: string): Promise<BackupSettings> {
return normalize((await readFile()).businesses[slug]);
}
export async function readAllBackupSettings(): Promise<Record<string, BackupSettings>> {
const file = await readFile();
return Object.fromEntries(Object.entries(file.businesses).map(([slug, raw]) => [slug, normalize(raw)]));
}
export interface BackupSettingsInput {
enabled: boolean;
frequency: BackupFrequency;
time: string;
weekday: number;
keep: number;
}
export function validateBackupSettingsInput(input: BackupSettingsInput): string | null {
if (!isValidBackupTime(input.time)) return "Enter the time as HH:MM (24-hour).";
if (input.frequency !== "daily" && input.frequency !== "weekly") return "Choose daily or weekly.";
if (!Number.isInteger(input.weekday) || input.weekday < 0 || input.weekday > 6) return "Choose a day of the week.";
if (!Number.isInteger(input.keep) || input.keep < 1 || input.keep > MAX_KEEP) return `Keep between 1 and ${MAX_KEEP} backups.`;
return null;
}
export function saveBackupSettings(slug: string, input: BackupSettingsInput, now: Date = new Date()): Promise<BackupSettings> {
return withMutex("backup-settings", async () => {
const file = await readFile();
const current = normalize(file.businesses[slug]);
const next: BackupSettings = {
...current,
enabled: input.enabled,
frequency: input.frequency,
time: input.time,
weekday: input.weekday,
keep: input.keep,
// The first scheduled run waits for the next scheduled time after enabling, rather than firing the moment the box is ticked ("Back up now" exists for that).
enabledAt: input.enabled && !current.enabled ? now.toISOString() : current.enabledAt,
};
file.businesses[slug] = next;
await writeJsonFileAtomic(backupSettingsPath(), file);
return next;
});
}
export function recordBackupOutcome(
slug: string,
outcome: { ok: boolean; message: string; at?: Date },
): Promise<void> {
return withMutex("backup-settings", async () => {
const file = await readFile();
const current = normalize(file.businesses[slug]);
const at = (outcome.at ?? new Date()).toISOString();
file.businesses[slug] = {
...current,
lastAttemptAt: at,
lastSuccessAt: outcome.ok ? at : current.lastSuccessAt,
lastStatus: outcome.ok ? "ok" : "failed",
lastMessage: outcome.message.slice(0, 500),
};
await writeJsonFileAtomic(backupSettingsPath(), file);
});
}
/** Marks that a scheduled run was STARTED (so a failing backup retries at the next scheduled time, not every scheduler tick). */
export function recordBackupAttempt(slug: string, at: Date = new Date()): Promise<void> {
return withMutex("backup-settings", async () => {
const file = await readFile();
file.businesses[slug] = { ...normalize(file.businesses[slug]), lastAttemptAt: at.toISOString() };
await writeJsonFileAtomic(backupSettingsPath(), file);
});
}
/** The most recent scheduled moment at or before `now` (Manager-local time). */
export function lastScheduledOccurrence(settings: Pick<BackupSettings, "frequency" | "time" | "weekday">, now: Date): Date {
const [hours, minutes] = settings.time.split(":").map(Number) as [number, number];
const candidate = new Date(now);
candidate.setHours(hours, minutes, 0, 0);
if (candidate.getTime() > now.getTime()) candidate.setDate(candidate.getDate() - 1);
if (settings.frequency === "weekly") {
while (candidate.getDay() !== settings.weekday) candidate.setDate(candidate.getDate() - 1);
}
return candidate;
}
/**
* Due when a scheduled moment has passed since the later of "last
* attempt" and "when it was enabled". Keyed on attempts, not successes:
* a backup that fails (disk full, MySQL down) is retried at the NEXT
* scheduled time and alerted on, instead of being re-run every few
* seconds all day.
*/
export function isBackupDue(settings: BackupSettings, now: Date): boolean {
if (!settings.enabled) return false;
const occurrence = lastScheduledOccurrence(settings, now);
const since = Math.max(
settings.lastAttemptAt ? Date.parse(settings.lastAttemptAt) : 0,
settings.enabledAt ? Date.parse(settings.enabledAt) : 0,
);
return occurrence.getTime() > since;
}
+136
View File
@@ -0,0 +1,136 @@
import { mkdir, readdir, rm, stat } from "node:fs/promises";
import path from "node:path";
import { BACKUP_KINDS, type BackupKind } from "./backup-manifest";
import { backupDir, backupsRoot } from "./manager-paths";
import { isValidSlug } from "./slug";
/**
* Where archives live and how they're named/pruned:
*
* STACKS_DIR/backups/<slug>/<slug>__<UTC stamp>__<kind>.7z
*
* `kind` decides retention (see pruneBackups): regular backups (scheduled
* + "Back up now") are pruned to the configured count; safety backups
* (taken automatically before an update or a replacing restore) keep the
* newest SAFETY_KEEP so they can never evict the regular ones you might be
* about to restore from; final backups (taken just before deleting a
* business) are never pruned automatically.
*/
export const DEFAULT_KEEP = 2;
export const MAX_KEEP = 14;
export const SAFETY_KEEP = 2;
const FILE_PATTERN = /^([a-z][a-z0-9-]{1,48}[a-z0-9])__(\d{8}T\d{6}Z)__(regular|safety|final)\.7z$/;
export interface BackupEntry {
slug: string;
fileName: string;
kind: BackupKind;
createdAt: string;
sizeBytes: number;
}
function stamp(date: Date): string {
return date.toISOString().replace(/[-:]/g, "").replace(/\.\d{3}Z$/, "Z");
}
function stampToIso(value: string): string {
const m = /^(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})(\d{2})Z$/.exec(value)!;
return `${m[1]}-${m[2]}-${m[3]}T${m[4]}:${m[5]}:${m[6]}Z`;
}
export function backupFileName(slug: string, kind: BackupKind, at: Date): string {
return `${slug}__${stamp(at)}__${kind}.7z`;
}
export function parseBackupFileName(fileName: string): { slug: string; kind: BackupKind; createdAt: string } | null {
const match = FILE_PATTERN.exec(fileName);
if (!match) return null;
const [, slug, rawStamp, kind] = match;
return { slug: slug!, kind: kind as BackupKind, createdAt: stampToIso(rawStamp!) };
}
/** Resolves a (slug, fileName) pair from untrusted input to an absolute path — rejects anything that isn't exactly a name this module produces. */
export function backupFilePath(slug: string, fileName: string): string {
const parsed = parseBackupFileName(fileName);
if (!isValidSlug(slug) || !parsed || parsed.slug !== slug) throw new Error("Invalid backup reference.");
return path.join(backupDir(slug), fileName);
}
export async function ensureBackupDir(slug: string): Promise<string> {
const dir = backupDir(slug);
await mkdir(dir, { recursive: true, mode: 0o700 });
return dir;
}
export async function listBackups(slug: string): Promise<BackupEntry[]> {
let files: string[];
try {
files = await readdir(backupDir(slug));
} catch {
return [];
}
const entries: BackupEntry[] = [];
for (const fileName of files) {
const parsed = parseBackupFileName(fileName);
if (!parsed || parsed.slug !== slug) continue;
try {
const info = await stat(path.join(backupDir(slug), fileName));
entries.push({ slug, fileName, kind: parsed.kind, createdAt: parsed.createdAt, sizeBytes: info.size });
} catch {
// vanished between readdir and stat
}
}
return entries.sort((a, b) => b.fileName.localeCompare(a.fileName) || b.createdAt.localeCompare(a.createdAt));
}
/** Every backup of every business — including businesses that have since been deleted (their archives stay). */
export async function listAllBackups(): Promise<BackupEntry[]> {
let dirs: string[];
try {
dirs = await readdir(backupsRoot());
} catch {
return [];
}
const all: BackupEntry[] = [];
for (const slug of dirs) {
if (!isValidSlug(slug)) continue;
all.push(...(await listBackups(slug)));
}
return all.sort((a, b) => b.createdAt.localeCompare(a.createdAt));
}
/** Newest-first within the kind; everything past `keep` is deleted. Never touches other kinds. Returns the removed file names. */
export async function pruneBackups(slug: string, kind: BackupKind, keep: number): Promise<string[]> {
if (kind === "final") return [];
const safeKeep = Math.max(1, Math.floor(keep));
const ofKind = (await listBackups(slug)).filter((b) => b.kind === kind);
const removed: string[] = [];
for (const old of ofKind.slice(safeKeep)) {
await rm(backupFilePath(slug, old.fileName), { force: true });
removed.push(old.fileName);
}
return removed;
}
export async function deleteBackup(slug: string, fileName: string): Promise<void> {
await rm(backupFilePath(slug, fileName), { force: true });
}
/** A backup that crashed mid-write leaves `*.partial` files; under the business lock nothing else can be writing, so they're all dead. */
export async function removePartialFiles(slug: string): Promise<void> {
let files: string[];
try {
files = await readdir(backupDir(slug));
} catch {
return;
}
for (const file of files) {
if (file.endsWith(".partial")) await rm(path.join(backupDir(slug), file), { force: true });
}
}
export function isBackupKind(value: string): value is BackupKind {
return (BACKUP_KINDS as readonly string[]).includes(value);
}
+196
View File
@@ -0,0 +1,196 @@
import { chmod, copyFile, mkdir, open, rename, rm, stat, writeFile } from "node:fs/promises";
import path from "node:path";
import { BACKUP_MEMBERS, buildManifest, type BackupKind } from "./backup-manifest";
import { readBackupSettings } from "./backup-settings";
import { backupFileName, ensureBackupDir, pruneBackups, removePartialFiles, SAFETY_KEEP } from "./backup-store";
import { containerExists } from "./docker";
import { mediaCopyOutCommand, mediaVolumeTarCommand, mysqlQueryCommand, mysqldumpCommand } from "./docker-commands";
import { readEnvFile, readEnvValue } from "./env-file";
import { readBackupPassphrase } from "./global-settings";
import { stagingRoot } from "./manager-paths";
import { runCapture, runToFile } from "./process";
import { randomHex } from "./random";
import { sevenZipCreate, sevenZipTest } from "./sevenzip";
import { stackDir, stackExists, stacksRoot } from "./stacks";
/**
* Builds one encrypted archive for a business:
*
* manifest.json what this is (slug, domain, db names, last migration…)
* db.sql mysqldump, streamed straight to disk
* media.tar /app/storage/media, streamed out of the container/volume
* config/.env, config/compose.yaml, config/site.caddy
*
* packaged with `7zz a -mhe=on -p…`, written as `<name>.7z.partial`,
* verified with `7zz t`, and only then renamed into place — so a crash or
* a failed verify can never leave something that looks like a good
* backup. Does NOT take the business lock; callers (the backup job, or a
* restore's safety step) already hold it.
*/
export const BACKUP_STEPS = [
{ id: "dump", label: "Dump the database" },
{ id: "media", label: "Copy media files" },
{ id: "config", label: "Collect configuration" },
{ id: "archive", label: "Encrypt and compress" },
{ id: "verify", label: "Verify the archive" },
{ id: "finish", label: "Save and prune old backups" },
] as const;
export interface BackupRunContext {
log(line: string): void;
/** Optional: wraps each phase as a job step. Absent when nested inside another job. */
step?<T>(stepId: string, fn: () => Promise<T>): Promise<T>;
}
export interface BackupResult {
slug: string;
fileName: string;
sizeBytes: number;
kind: BackupKind;
pruned: string[];
}
function runStep<T>(ctx: BackupRunContext, id: string, fn: () => Promise<T>): Promise<T> {
return ctx.step ? ctx.step(id, fn) : fn();
}
async function assertDumpComplete(dumpPath: string): Promise<void> {
const info = await stat(dumpPath);
const handle = await open(dumpPath, "r");
try {
const length = Math.min(512, info.size);
const buffer = Buffer.alloc(length);
await handle.read(buffer, 0, length, info.size - length);
if (!buffer.toString("utf8").includes("-- Dump completed")) {
throw new Error("The database dump looks incomplete (no completion marker at the end).");
}
} finally {
await handle.close();
}
}
async function readLastMigration(databaseName: string): Promise<string | null> {
try {
const { stdout } = await runCapture(
mysqlQueryCommand(databaseName, "SELECT migration_name FROM _prisma_migrations WHERE finished_at IS NOT NULL ORDER BY migration_name DESC LIMIT 1"),
{ maxBytes: 4096, timeoutMs: 30_000 },
);
return stdout.trim() || null;
} catch {
return null;
}
}
export async function performBackup(slug: string, kind: BackupKind, ctx: BackupRunContext, now: Date = new Date()): Promise<BackupResult> {
if (!(await stackExists(slug))) throw new Error(`No such business: ${slug}`);
const dir = stackDir(slug);
const envContent = await readEnvFile(path.join(dir, ".env"));
if (!envContent) throw new Error(`"${slug}" has no .env — it was never fully provisioned, so there is nothing to back up.`);
const databaseName = readEnvValue(envContent, "DATABASE_NAME");
const databaseUser = readEnvValue(envContent, "DATABASE_USER");
const domain = readEnvValue(envContent, "DOMAIN");
if (!databaseName || !databaseUser || !domain) throw new Error(`"${slug}"'s .env is missing DATABASE_NAME/DATABASE_USER/DOMAIN.`);
const passphrase = await readBackupPassphrase();
if (!passphrase) {
throw new Error("No backup passphrase exists yet. Enable backups for a business (or use Back up now) from the Backups page so one is generated and shown to you.");
}
const outDir = await ensureBackupDir(slug);
await removePartialFiles(slug);
const finalName = backupFileName(slug, kind, now);
const partialPath = path.join(outDir, `${finalName}.partial`);
const finalPath = path.join(outDir, finalName);
const stagingDir = path.join(stagingRoot(), `backup-${slug}-${randomHex(4)}`);
await mkdir(path.join(stagingDir, "config"), { recursive: true, mode: 0o700 });
try {
const dumpPath = path.join(stagingDir, BACKUP_MEMBERS.database);
await runStep(ctx, "dump", async () => {
ctx.log(`Dumping database ${databaseName}…`);
await runToFile(mysqldumpCommand(databaseName), dumpPath);
await assertDumpComplete(dumpPath);
ctx.log(`Database dump written (${(await stat(dumpPath)).size} bytes).`);
});
const mediaPath = path.join(stagingDir, BACKUP_MEMBERS.media);
await runStep(ctx, "media", async () => {
const container = `next-chair-${slug}`;
if (await containerExists(container)) {
ctx.log("Copying media from the running container…");
await runToFile(mediaCopyOutCommand(container), mediaPath);
} else {
ctx.log("Container isn't present (stack stopped) — reading media from its volume…");
await runToFile(mediaVolumeTarCommand(slug), mediaPath);
}
ctx.log(`Media archive written (${(await stat(mediaPath)).size} bytes).`);
});
await runStep(ctx, "config", async () => {
await copyFile(path.join(dir, ".env"), path.join(stagingDir, BACKUP_MEMBERS.env));
await copyFile(path.join(dir, "compose.yaml"), path.join(stagingDir, BACKUP_MEMBERS.compose));
let hasCaddySite = false;
try {
await copyFile(path.join(stacksRoot(), "infra", "sites", `${slug}.caddy`), path.join(stagingDir, BACKUP_MEMBERS.caddy));
hasCaddySite = true;
} catch {
ctx.log("No Caddy site file found for this business — skipping it.");
}
const lastMigration = await readLastMigration(databaseName);
const manifest = buildManifest({
slug,
name: readEnvValue(envContent, "BUSINESS_NAME"),
domain,
databaseName,
databaseUser,
createdAt: now.toISOString(),
kind,
lastMigration,
appImage: process.env.NEXTCHAIR_IMAGE ?? null,
hasCaddySite,
});
await writeFile(path.join(stagingDir, BACKUP_MEMBERS.manifest), JSON.stringify(manifest, null, 2), { encoding: "utf8", mode: 0o600 });
ctx.log(`Manifest written (last migration: ${lastMigration ?? "unknown"}).`);
});
await runStep(ctx, "archive", async () => {
ctx.log("Encrypting and compressing…");
await rm(partialPath, { force: true });
await sevenZipCreate({
archivePath: partialPath,
cwd: stagingDir,
entries: [BACKUP_MEMBERS.manifest, BACKUP_MEMBERS.database, BACKUP_MEMBERS.media, "config"],
passphrase,
});
});
await runStep(ctx, "verify", async () => {
ctx.log("Verifying the archive (7zz t)…");
await sevenZipTest(partialPath, passphrase);
});
return await runStep(ctx, "finish", async () => {
await rename(partialPath, finalPath);
await chmod(finalPath, 0o600); // 7zz honours the umask; the archive (encrypted or not) should be owner-only
const sizeBytes = (await stat(finalPath)).size;
let pruned: string[] = [];
if (kind === "regular") {
const { keep } = await readBackupSettings(slug);
pruned = await pruneBackups(slug, "regular", keep);
} else if (kind === "safety") {
pruned = await pruneBackups(slug, "safety", SAFETY_KEEP);
}
if (pruned.length > 0) ctx.log(`Removed ${pruned.length} old backup(s): ${pruned.join(", ")}`);
ctx.log(`Backup saved: ${finalName} (${sizeBytes} bytes).`);
return { slug, fileName: finalName, sizeBytes, kind, pruned };
});
} catch (error) {
await rm(partialPath, { force: true });
throw error;
} finally {
await rm(stagingDir, { recursive: true, force: true });
}
}
+127
View File
@@ -0,0 +1,127 @@
import type { CommandSpec } from "./process";
/**
* Pure command builders for every `docker` call the backup/restore code
* streams (see process.ts). Kept separate from docker.ts (which actually
* runs `docker compose`) so the exact argument arrays are plain,
* unit-testable values: no shell, no string interpolation into a command
* line, and the MySQL root password travels as the MYSQL_PWD environment
* variable of the docker CLI (`-e MYSQL_PWD` with no value forwards the
* client's own env var into the exec) — never as an argument, so it never
* shows up in a process listing.
*/
export const MEDIA_MOUNT_PATH = "/app/storage/media";
/** Uid/gid the business image's `nextchair` user runs as (see the main Dockerfile). */
export const APP_UID_GID = "1001:1001";
export function mysqlContainerName(): string {
return process.env.MYSQL_CONTAINER_NAME ?? "next-chair-mysql";
}
export function appImage(): string {
return process.env.NEXTCHAIR_IMAGE ?? "nextchair-app:latest";
}
export function mediaVolumeName(slug: string): string {
return `next-chair-media-${slug}`;
}
function mysqlEnv(): Record<string, string> {
const password = process.env.MYSQL_ROOT_PASSWORD;
if (!password) throw new Error("MYSQL_ROOT_PASSWORD must be set.");
return { MYSQL_PWD: password };
}
export function mysqldumpCommand(databaseName: string): CommandSpec {
return {
command: "docker",
args: [
"exec",
"-e",
"MYSQL_PWD",
mysqlContainerName(),
"mysqldump",
"-uroot",
"--single-transaction",
"--routines",
"--triggers",
"--no-tablespaces",
"--hex-blob",
"--default-character-set=utf8mb4",
databaseName,
],
env: mysqlEnv(),
};
}
export function mysqlImportCommand(databaseName: string): CommandSpec {
return {
command: "docker",
args: ["exec", "-i", "-e", "MYSQL_PWD", mysqlContainerName(), "mysql", "-uroot", "--default-character-set=utf8mb4", databaseName],
env: mysqlEnv(),
};
}
/** `-N -B`: no header, tab-separated — for reading a single value back. */
export function mysqlQueryCommand(databaseName: string, sql: string): CommandSpec {
return {
command: "docker",
args: ["exec", "-e", "MYSQL_PWD", mysqlContainerName(), "mysql", "-uroot", "-N", "-B", databaseName, "-e", sql],
env: mysqlEnv(),
};
}
/** tar stream of /app/storage/media (entries are prefixed `media/`) from a business's existing container. */
export function mediaCopyOutCommand(containerName: string): CommandSpec {
return { command: "docker", args: ["cp", `${containerName}:${MEDIA_MOUNT_PATH}`, "-"] };
}
/** Same tar layout as mediaCopyOutCommand, read straight from the named volume — used when the container has been removed (a stopped stack is `compose down`, which deletes it). */
export function mediaVolumeTarCommand(slug: string): CommandSpec {
return {
command: "docker",
args: [
"run",
"--rm",
"-v",
`${mediaVolumeName(slug)}:${MEDIA_MOUNT_PATH}:ro`,
"--entrypoint",
"tar",
appImage(),
"-c",
"-C",
"/app/storage",
"media",
],
};
}
/** Unpacks the tar stream on stdin into the (fresh) media volume and hands it to the app user. The shell script is a constant — nothing is interpolated into it. */
export function mediaRestoreCommand(slug: string): CommandSpec {
return {
command: "docker",
args: [
"run",
"--rm",
"-i",
"--user",
"0",
"-v",
`${mediaVolumeName(slug)}:${MEDIA_MOUNT_PATH}`,
"--entrypoint",
"sh",
appImage(),
"-c",
`tar -x -C /app/storage && chown -R ${APP_UID_GID} ${MEDIA_MOUNT_PATH}`,
],
};
}
export function imageMigrationsCommand(): CommandSpec {
return { command: "docker", args: ["run", "--rm", "--entrypoint", "ls", appImage(), "/app/prisma/migrations"] };
}
export function containerExistsCommand(containerName: string): CommandSpec {
return { command: "docker", args: ["container", "inspect", "--format", "{{.Id}}", containerName] };
}
+26
View File
@@ -1,5 +1,7 @@
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { containerExistsCommand, imageMigrationsCommand } from "./docker-commands";
import { runCapture } from "./process";
const execFileAsync = promisify(execFile);
@@ -35,6 +37,11 @@ export async function composeUp(stackDir: string, options: { forceRecreate?: boo
await runDockerCompose(stackDir, args);
}
/** Creates the stack's container(s) and volume(s) without starting them — lets a restore populate the media volume before the app's first boot. */
export async function composeCreate(stackDir: string): Promise<void> {
await runDockerCompose(stackDir, ["create"]);
}
export async function composeDown(stackDir: string, options: { removeVolumes?: boolean } = {}): Promise<void> {
const args = ["down"];
if (options.removeVolumes) args.push("-v");
@@ -89,3 +96,22 @@ export async function reloadCaddy(): Promise<void> {
const containerName = process.env.CADDY_CONTAINER_NAME ?? "next-chair-caddy";
await execFileAsync("docker", ["exec", containerName, "caddy", "reload", "--config", "/etc/caddy/Caddyfile"]);
}
export async function containerExists(containerName: string): Promise<boolean> {
try {
await runCapture(containerExistsCommand(containerName), { maxBytes: 4096 });
return true;
} catch {
return false;
}
}
/** Migration folder names baked into the shared app image (sorted) — read by running `ls` in a throwaway container, so it works on a fresh server with no business running. */
export async function listImageMigrations(): Promise<string[]> {
const { stdout } = await runCapture(imageMigrationsCommand(), { maxBytes: 256 * 1024, timeoutMs: 120_000 });
return stdout
.split("\n")
.map((line) => line.trim())
.filter((line) => /^\d{14}_/.test(line))
.sort();
}
+122 -1
View File
@@ -2,6 +2,7 @@ import { randomBytes, createCipheriv, createDecipheriv } from "node:crypto";
import { mkdir, readFile, writeFile } from "node:fs/promises";
import path from "node:path";
import webpush from "web-push";
import { withMutex } from "./singleton";
import { stacksRoot } from "./stacks";
/**
@@ -61,11 +62,46 @@ interface StoredGoogleCalendarSettings {
encryptedClientSecret: EncryptedPayload | null;
}
/**
* Operator-facing monitoring + scheduler knobs (Settings > Monitoring).
* Not secret, so stored and returned in plain.
*/
export interface OperatorSettings {
/** Where health / backup alerts go. Blank = alerting is off (nothing is emailed). */
alertEmail: string;
alertOnHealth: boolean;
alertOnRecovery: boolean;
/** While a business is still down/degraded, repeat the alert once a day. */
dailyReminder: boolean;
alertOnBackupFailure: boolean;
/** How often the built-in scheduler calls each business's notification cron endpoint. */
notificationsIntervalSeconds: number;
healthIntervalMinutes: number;
}
export const DEFAULT_OPERATOR_SETTINGS: OperatorSettings = {
alertEmail: "",
alertOnHealth: true,
alertOnRecovery: true,
dailyReminder: true,
alertOnBackupFailure: true,
notificationsIntervalSeconds: 30,
healthIntervalMinutes: 3,
};
/** Persisted shape — the passphrase only ever holds the encrypted envelope. One passphrase protects every business's backups, so a single value is enough to restore on a fresh server. */
interface StoredBackupSettings {
encryptedPassphrase: EncryptedPayload | null;
createdAt: string | null;
}
interface StoredGlobalSettings {
email: StoredEmailSettings | null;
push: StoredPushSettings | null;
superAdmin: SuperAdminSettings | null;
googleCalendar: StoredGoogleCalendarSettings | null;
operator: OperatorSettings | null;
backups: StoredBackupSettings | null;
}
export interface LastTestResult {
@@ -157,10 +193,12 @@ async function readStoredSettings(): Promise<StoredGlobalSettings> {
push: parsed.push ?? null,
superAdmin: parsed.superAdmin ?? null,
googleCalendar: parsed.googleCalendar ?? null,
operator: parsed.operator ?? null,
backups: parsed.backups ?? null,
};
} catch {
// No file yet (first run) or unreadable — treat as "nothing configured" rather than throwing, so the Settings page can always render.
return { email: null, push: null, superAdmin: null, googleCalendar: null };
return { email: null, push: null, superAdmin: null, googleCalendar: null, operator: null, backups: null };
}
}
@@ -490,3 +528,86 @@ export async function saveGoogleCalendarSettings(input: GoogleCalendarSettingsIn
return { clientId: next.clientId, clientSecret: resolvedSecret };
}
// ── Operator monitoring / scheduler ──────────────────────────────────────
function clampInt(value: unknown, min: number, max: number, fallback: number): number {
const n = typeof value === "number" ? value : Number(value);
if (!Number.isFinite(n)) return fallback;
return Math.min(max, Math.max(min, Math.round(n)));
}
export async function readOperatorSettings(): Promise<OperatorSettings> {
const { operator } = await readStoredSettings();
// Merge over defaults so a settings file written before a field existed still yields a complete object.
const merged = { ...DEFAULT_OPERATOR_SETTINGS, ...(operator ?? {}) };
return {
...merged,
notificationsIntervalSeconds: clampInt(merged.notificationsIntervalSeconds, 10, 3600, 30),
healthIntervalMinutes: clampInt(merged.healthIntervalMinutes, 1, 60, 3),
};
}
export async function saveOperatorSettings(input: OperatorSettings): Promise<OperatorSettings> {
const current = await readStoredSettings();
const next: OperatorSettings = {
alertEmail: input.alertEmail.trim(),
alertOnHealth: input.alertOnHealth,
alertOnRecovery: input.alertOnRecovery,
dailyReminder: input.dailyReminder,
alertOnBackupFailure: input.alertOnBackupFailure,
notificationsIntervalSeconds: clampInt(input.notificationsIntervalSeconds, 10, 3600, 30),
healthIntervalMinutes: clampInt(input.healthIntervalMinutes, 1, 60, 3),
};
await writeStoredSettings({ ...current, operator: next });
return next;
}
// ── Backup passphrase ────────────────────────────────────────────────────
export async function hasBackupPassphrase(): Promise<boolean> {
const { backups } = await readStoredSettings();
return Boolean(backups?.encryptedPassphrase);
}
/** Server-side only — the decrypted passphrase used for scheduled/manual archives. Null until one has been generated. */
export async function readBackupPassphrase(): Promise<string | null> {
const { backups } = await readStoredSettings();
return backups?.encryptedPassphrase ? decrypt(backups.encryptedPassphrase) : null;
}
export async function backupPassphraseCreatedAt(): Promise<string | null> {
const { backups } = await readStoredSettings();
return backups?.createdAt ?? null;
}
/**
* Returns the stored passphrase, generating (and encrypting) one on first
* use. `created` is true only on that first call — the one moment the UI
* shows the plaintext to the operator so they can save it somewhere safe;
* afterwards it is never displayed again (a fresh server needs it typed
* in at restore time).
*/
export function ensureBackupPassphrase(): Promise<{ passphrase: string; created: boolean }> {
// Mutex: a scheduled backup and a manual one racing on a fresh install must not each mint a different passphrase.
return withMutex("backup-passphrase", async () => {
const existing = await readBackupPassphrase();
if (existing) return { passphrase: existing, created: false };
return { passphrase: await storeNewBackupPassphrase(), created: true };
});
}
/** Replaces the passphrase (older archives still need the old one). Returns the new plaintext, shown once. */
export async function rotateBackupPassphrase(): Promise<string> {
return storeNewBackupPassphrase();
}
async function storeNewBackupPassphrase(): Promise<string> {
const passphrase = randomBytes(18).toString("base64url");
const current = await readStoredSettings();
await writeStoredSettings({
...current,
backups: { encryptedPassphrase: encrypt(passphrase), createdAt: new Date().toISOString() },
});
return passphrase;
}
+269
View File
@@ -0,0 +1,269 @@
import path from "node:path";
import { readEnvFile, readEnvValue } from "./env-file";
import { readOperatorSettings, type OperatorSettings } from "./global-settings";
import { fetchHealthSnapshot, type HealthSnapshot, type HealthVerdict } from "./health";
import { readJsonFile, writeJsonFileAtomic } from "./json-file";
import { getActiveLock } from "./locks";
import { healthStatePath } from "./manager-paths";
import { sendOperatorEmail } from "./operator-email";
import { withMutex } from "./singleton";
import { stackDir } from "./stacks";
/**
* Per-business health state machine (ok / degraded / down) with the
* alerting rules from the plan:
*
* - two consecutive bad checks before the first alert (one failed probe
* during a restart or a slow request is not an incident);
* - an email on every state change and on recovery;
* - while a business is still bad, one reminder per day;
* - businesses being stopped on purpose, or mid-update/restore, are not
* probed at all.
*
* `applyObservation` is a pure function (all the policy, unit-tested);
* `runHealthCheck` wires it to the probe, the state file and the email.
*/
export const BAD_THRESHOLD = 2;
export const RECOVER_THRESHOLD = 1;
export const REMINDER_INTERVAL_MS = 24 * 60 * 60 * 1000;
/** An alert that couldn't be emailed (SMTP down) is retried each check, then given up on after this long. */
const UNSENT_ALERT_TTL_MS = 24 * 60 * 60 * 1000;
export type HealthAlertType = "problem" | "changed" | "recovered" | "reminder";
export interface HealthAlert {
type: HealthAlertType;
slug: string;
from: HealthVerdict;
to: HealthVerdict;
reasons: string[];
/** When the current state began. */
since: string;
raisedAt: string;
}
export interface HealthRecord {
status: HealthVerdict;
since: string;
reasons: string[];
pending: { verdict: HealthVerdict; count: number } | null;
lastAlertAt: string | null;
lastCheckedAt: string | null;
unsentAlert: HealthAlert | null;
}
export function initialRecord(now: Date): HealthRecord {
const iso = now.toISOString();
return { status: "ok", since: iso, reasons: [], pending: null, lastAlertAt: null, lastCheckedAt: iso, unsentAlert: null };
}
export interface Observation {
verdict: HealthVerdict;
reasons: string[];
}
export function applyObservation(
slug: string,
previous: HealthRecord | undefined,
observation: Observation,
now: Date,
options: { dailyReminder: boolean },
): { next: HealthRecord; alert: HealthAlert | null } {
const prev = previous ?? initialRecord(now);
const iso = now.toISOString();
if (observation.verdict === prev.status) {
const next: HealthRecord = { ...prev, pending: null, reasons: observation.reasons, lastCheckedAt: iso };
// Baseline is the last alert, or — if none was ever delivered (alerting was off, SMTP down) — when the bad state began.
const baseline = Date.parse(prev.lastAlertAt ?? prev.since);
const reminderDue = prev.status !== "ok" && options.dailyReminder && now.getTime() - baseline >= REMINDER_INTERVAL_MS;
const alert: HealthAlert | null = reminderDue
? { type: "reminder", slug, from: prev.status, to: prev.status, reasons: observation.reasons, since: prev.since, raisedAt: iso }
: null;
return { next, alert };
}
const threshold = observation.verdict === "ok" ? RECOVER_THRESHOLD : BAD_THRESHOLD;
const count = prev.pending && prev.pending.verdict === observation.verdict ? prev.pending.count + 1 : 1;
if (count < threshold) {
return { next: { ...prev, pending: { verdict: observation.verdict, count }, lastCheckedAt: iso }, alert: null };
}
const type: HealthAlertType = observation.verdict === "ok" ? "recovered" : prev.status === "ok" ? "problem" : "changed";
const next: HealthRecord = {
...prev,
status: observation.verdict,
since: iso,
reasons: observation.reasons,
pending: null,
lastCheckedAt: iso,
};
return {
next,
// For a recovery, `since` is when the BAD period began (so the email can say how long it lasted).
alert: { type, slug, from: prev.status, to: observation.verdict, reasons: observation.reasons, since: type === "recovered" ? prev.since : iso, raisedAt: iso },
};
}
// ── State file ───────────────────────────────────────────────────────────
interface HealthStateFile {
businesses: Record<string, HealthRecord>;
/** Slugs deliberately stopped from the Manager — not probed, not alerted on. */
stopped: Record<string, string>;
}
async function readState(): Promise<HealthStateFile> {
const file = await readJsonFile<Partial<HealthStateFile>>(healthStatePath(), {});
return { businesses: file.businesses ?? {}, stopped: file.stopped ?? {} };
}
export async function readHealthRecords(): Promise<Record<string, HealthRecord>> {
return (await readState()).businesses;
}
/** Called by the Stop / Start actions so an intentional stop doesn't page anyone. */
export function setIntentionallyStopped(slug: string, stopped: boolean, now: Date = new Date()): Promise<void> {
return withMutex("health-state", async () => {
const state = await readState();
if (stopped) state.stopped[slug] = now.toISOString();
else {
delete state.stopped[slug];
// Back from a deliberate stop: judge it afresh.
delete state.businesses[slug];
}
await writeJsonFileAtomic(healthStatePath(), state);
});
}
// ── Alert email ──────────────────────────────────────────────────────────
export function buildAlertEmail(alert: HealthAlert, label: string, domain: string | null): { subject: string; text: string } {
const where = domain ? `${label} (${domain})` : label;
const reasons = alert.reasons.length > 0 ? alert.reasons.map((r) => ` - ${r}`).join("\n") : " (no detail reported)";
switch (alert.type) {
case "recovered":
return {
subject: `${where} has recovered`,
text: `${where} is healthy again (was ${alert.from} since ${alert.since}).\n\nRecovered at: ${alert.raisedAt}\n`,
};
case "reminder":
return {
subject: `Still ${alert.to.toUpperCase()}: ${where}`,
text: `${where} has been ${alert.to} since ${alert.since}.\n\nCurrent reasons:\n${reasons}\n\nThis is the daily reminder while it stays unresolved.\n`,
};
case "changed":
return {
subject: `${where} is now ${alert.to.toUpperCase()} (was ${alert.from})`,
text: `${where} changed from ${alert.from} to ${alert.to}.\n\nReasons:\n${reasons}\n\nTime: ${alert.raisedAt}\n`,
};
default:
return {
subject: `${where} is ${alert.to.toUpperCase()}`,
text: `${where} failed ${BAD_THRESHOLD} consecutive health checks and is now ${alert.to}.\n\nReasons:\n${reasons}\n\nTime: ${alert.raisedAt}\n`,
};
}
}
function shouldSend(alert: HealthAlert, settings: OperatorSettings): boolean {
if (!settings.alertEmail) return false;
if (alert.type === "recovered") return settings.alertOnRecovery;
if (alert.type === "reminder") return settings.alertOnHealth && settings.dailyReminder;
return settings.alertOnHealth;
}
export interface HealthCheckOutcome {
skipped?: string;
record?: HealthRecord;
alertSent?: HealthAlertType;
}
export interface HealthCheckDeps {
probe: (slug: string) => Promise<HealthSnapshot>;
sendEmail: (subject: string, text: string) => Promise<{ ok: boolean; message: string }>;
now: () => Date;
isBusy: (slug: string) => Promise<boolean>;
readSettings: () => Promise<OperatorSettings>;
describe: (slug: string) => Promise<{ label: string; domain: string | null }>;
}
async function describeBusiness(slug: string): Promise<{ label: string; domain: string | null }> {
const env = await readEnvFile(path.join(stackDir(slug), ".env"));
return { label: (env && readEnvValue(env, "BUSINESS_NAME")) || slug, domain: env ? readEnvValue(env, "DOMAIN") : null };
}
function defaultDeps(): HealthCheckDeps {
return {
probe: fetchHealthSnapshot,
sendEmail: sendOperatorEmail,
now: () => new Date(),
isBusy: async (slug) => (await getActiveLock(slug)) !== null,
readSettings: readOperatorSettings,
describe: describeBusiness,
};
}
export async function runHealthCheck(slug: string, overrides: Partial<HealthCheckDeps> = {}): Promise<HealthCheckOutcome> {
const deps = { ...defaultDeps(), ...overrides };
const stateBefore = await readState();
if (slug in stateBefore.stopped) return { skipped: "intentionally stopped" };
if (await deps.isBusy(slug)) return { skipped: "a job holds this business (update/backup/restore)" };
// Probe outside the state mutex — it can take seconds.
const snapshot = await deps.probe(slug);
const settings = await deps.readSettings();
const now = deps.now();
const { record, toSend } = await withMutex("health-state", async () => {
const state = await readState();
const { next, alert } = applyObservation(slug, state.businesses[slug], { verdict: snapshot.verdict, reasons: snapshot.reasons }, now, {
dailyReminder: settings.dailyReminder,
});
// A previous alert that never got emailed goes out first (unless it's gone stale).
const carried = state.businesses[slug]?.unsentAlert ?? null;
const carriedFresh = carried && now.getTime() - Date.parse(carried.raisedAt) < UNSENT_ALERT_TTL_MS ? carried : null;
const pick = alert ?? carriedFresh;
next.unsentAlert = alert ? null : carriedFresh;
state.businesses[slug] = next;
await writeJsonFileAtomic(healthStatePath(), state);
return { record: next, toSend: pick };
});
if (!toSend || !shouldSend(toSend, settings)) {
if (toSend) await clearUnsent(slug, toSend);
return { record };
}
const { label, domain } = await deps.describe(slug);
const email = buildAlertEmail(toSend, label, domain);
const result = await deps.sendEmail(email.subject, email.text);
await withMutex("health-state", async () => {
const state = await readState();
const current = state.businesses[slug];
if (!current) return;
if (result.ok) {
current.lastAlertAt = now.toISOString();
current.unsentAlert = null;
} else {
current.unsentAlert = toSend;
}
await writeJsonFileAtomic(healthStatePath(), state);
});
return { record, alertSent: result.ok ? toSend.type : undefined };
}
async function clearUnsent(slug: string, alert: HealthAlert): Promise<void> {
await withMutex("health-state", async () => {
const state = await readState();
const current = state.businesses[slug];
if (current?.unsentAlert?.raisedAt === alert.raisedAt) {
current.unsentAlert = null;
await writeJsonFileAtomic(healthStatePath(), state);
}
});
}
+158
View File
@@ -0,0 +1,158 @@
/**
* One probe of a business's own `GET /api/health` over the internal
* Docker network (the same `next-chair-<slug>:3000` host Caddy proxies
* to). The main app will later add optional `queue` / `cron` fields and a
* `status` of ok|degraded; they are consumed when present and, when
* absent, a reachable 200 is simply "healthy" — so this keeps working
* against an app image that predates them.
*/
export interface QueueHealth {
stalePendingCount: number | null;
oldestPendingAgeSeconds: number | null;
stuckProcessingCount: number | null;
failedLast24h: number | null;
}
export type HealthVerdict = "ok" | "degraded" | "down";
export interface HealthSnapshot {
verdict: HealthVerdict;
/** Human-readable reasons — empty when ok. Never includes response bodies verbatim. */
reasons: string[];
httpStatus: number | null;
installed: boolean | null;
installedAt: string | null;
lastCronRunAt: string | null;
queue: QueueHealth | null;
}
const PROBE_TIMEOUT_MS = 5000;
function num(value: unknown): number | null {
return typeof value === "number" && Number.isFinite(value) ? value : null;
}
function str(value: unknown): string | null {
return typeof value === "string" && value ? value : null;
}
function record(value: unknown): Record<string, unknown> | null {
return value && typeof value === "object" && !Array.isArray(value) ? (value as Record<string, unknown>) : null;
}
/** Pure: turns a parsed /api/health body (or a failure) into a verdict. Exported for tests. */
export function classifyHealthBody(httpStatus: number, body: unknown): HealthSnapshot {
const base: HealthSnapshot = {
verdict: "ok",
reasons: [],
httpStatus,
installed: null,
installedAt: null,
lastCronRunAt: null,
queue: null,
};
if (httpStatus !== 200) {
return { ...base, verdict: "down", reasons: [`/api/health returned HTTP ${httpStatus}`] };
}
const obj = record(body);
if (!obj) return { ...base, verdict: "down", reasons: ["/api/health did not return JSON"] };
const installed = typeof obj.installed === "boolean" ? obj.installed : null;
const cron = record(obj.cron);
const queueObj = record(obj.queue);
const queue: QueueHealth | null = queueObj
? {
stalePendingCount: num(queueObj.stalePendingCount ?? queueObj.stalePending),
oldestPendingAgeSeconds: num(queueObj.oldestPendingAgeSeconds ?? queueObj.oldestPendingAge),
stuckProcessingCount: num(queueObj.stuckProcessingCount ?? queueObj.stuckProcessing),
failedLast24h: num(queueObj.failedLast24h ?? queueObj.failed24h),
}
: null;
const reasons: string[] = [];
const rawReasons = obj.degradedReasons;
if (Array.isArray(rawReasons)) {
for (const reason of rawReasons) if (typeof reason === "string" && reason.trim()) reasons.push(reason.trim().slice(0, 200));
}
const status = typeof obj.status === "string" ? obj.status.toLowerCase() : "";
// The current app reports "healthy"/"unhealthy"; the planned fields use ok|degraded.
const reportedDegraded = status === "degraded" || reasons.length > 0;
if (reportedDegraded && reasons.length === 0) reasons.push("the app reports itself degraded");
const verdict: HealthVerdict = status === "unhealthy" ? "down" : reportedDegraded ? "degraded" : "ok";
if (verdict === "down" && reasons.length === 0) reasons.push("the app reports itself unhealthy");
return {
...base,
verdict,
reasons,
installed,
installedAt: str(obj.installedAt),
lastCronRunAt: str(cron?.lastCronRunAt) ?? str(obj.lastCronRunAt),
queue,
};
}
export async function fetchHealthSnapshot(slug: string): Promise<HealthSnapshot> {
try {
const response = await fetch(`http://next-chair-${slug}:3000/api/health`, {
cache: "no-store",
signal: AbortSignal.timeout(PROBE_TIMEOUT_MS),
});
let body: unknown = null;
try {
body = await response.json();
} catch {
body = null;
}
return classifyHealthBody(response.status, body);
} catch {
return {
verdict: "down",
reasons: ["couldn't reach the app (connection refused, DNS failure or timeout)"],
httpStatus: null,
installed: null,
installedAt: null,
lastCronRunAt: null,
queue: null,
};
}
}
export interface WaitForHealthOptions {
timeoutMs?: number;
intervalMs?: number;
/** Restore: the setup wizard must NOT appear, so wait for installed:true specifically. */
requireInstalled?: boolean;
onAttempt?: (snapshot: HealthSnapshot) => void;
probe?: (slug: string) => Promise<HealthSnapshot>;
sleep?: (ms: number) => Promise<void>;
}
/** Polls until the app answers 200 (and, if asked, reports installed:true). Throws a descriptive error on timeout. */
export async function waitForHealth(slug: string, options: WaitForHealthOptions = {}): Promise<HealthSnapshot> {
const timeoutMs = options.timeoutMs ?? 3 * 60 * 1000;
const intervalMs = options.intervalMs ?? 3000;
const probe = options.probe ?? fetchHealthSnapshot;
const sleep = options.sleep ?? ((ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)));
const deadline = Date.now() + timeoutMs;
let last: HealthSnapshot | null = null;
for (;;) {
last = await probe(slug);
options.onAttempt?.(last);
const reachable = last.httpStatus === 200;
if (reachable && (!options.requireInstalled || last.installed === true)) return last;
if (Date.now() >= deadline) break;
await sleep(intervalMs);
}
if (last?.httpStatus === 200 && options.requireInstalled) {
throw new Error(`"${slug}" is up but reports installed:false — the setup wizard would appear. The restored database may be empty or wrong.`);
}
throw new Error(`"${slug}" did not become healthy within ${Math.round(timeoutMs / 1000)}s${last?.reasons.length ? ` (${last.reasons.join("; ")})` : ""}.`);
}
+9
View File
@@ -0,0 +1,9 @@
/** Shared result shape for server actions that start a background job (kept out of the "use server" files, which may only export async functions). */
export interface JobStartState {
error: string | null;
jobId: string | null;
/** Plaintext backup passphrase — present only in the single response that just generated it. Shown once, never stored in a job record. */
newPassphrase: string | null;
}
export const NO_JOB: JobStartState = { error: null, jobId: null, newPassphrase: null };
+270
View File
@@ -0,0 +1,270 @@
import { readdir, readFile, rm } from "node:fs/promises";
import path from "node:path";
import { writeJsonFileAtomic } from "./json-file";
import { acquireLock, LOCK_HEARTBEAT_MS, releaseLock, releaseLockIfOwnedBy, touchLock } from "./locks";
import { jobsDir } from "./manager-paths";
import { randomHex } from "./random";
import { globalSingleton, withMutex } from "./singleton";
/**
* Background job runner. A job is started from a request (server action),
* runs on in this process after the request returns, and writes its whole
* state to STACKS_DIR/manager/jobs/<id>.json after every change — the UI
* polls GET /api/jobs/<id> and renders the steps and log tail. State in a
* file (not memory) so any server bundle can serve the poll, and a
* finished job's outcome survives a Manager restart.
*
* Every job holds the per-business lock (locks.ts) for its whole run, so
* Update / Backup / Restore / Delete can't overlap on one business.
*/
export type JobKind = "update" | "backup" | "restore" | "delete";
export type JobStatus = "running" | "succeeded" | "failed";
export type StepStatus = "pending" | "running" | "done" | "failed" | "skipped";
export interface JobStep {
id: string;
label: string;
status: StepStatus;
startedAt: string | null;
finishedAt: string | null;
detail: string | null;
}
export type JobResult = Record<string, string | number | boolean | null>;
export interface JobRecord {
id: string;
kind: JobKind;
slug: string;
title: string;
status: JobStatus;
createdAt: string;
finishedAt: string | null;
steps: JobStep[];
logTail: string[];
error: string | null;
result: JobResult | null;
}
export interface JobContext {
jobId: string;
log(line: string): void;
/** Runs `fn` as the named step: marks it running, then done (or failed, rethrowing). */
step<T>(stepId: string, fn: () => Promise<T>): Promise<T>;
skip(stepId: string, detail?: string): void;
setResult(result: JobResult): void;
}
const LOG_TAIL_LINES = 200;
const LOG_LINE_MAX = 500;
const JOB_ID_PATTERN = /^[a-z0-9]{6,12}-[a-f0-9]{8}$/;
export function isValidJobId(id: string): boolean {
return JOB_ID_PATTERN.test(id);
}
function jobPath(id: string): string {
return path.join(jobsDir(), `${id}.json`);
}
/** Ids of jobs running in THIS process — recovery must never "fail" a job that is genuinely still going. */
function runningJobIds(): Set<string> {
return globalSingleton("running_job_ids", () => new Set<string>());
}
export async function readJob(id: string): Promise<JobRecord | null> {
if (!isValidJobId(id)) return null;
try {
return JSON.parse(await readFile(jobPath(id), "utf8")) as JobRecord;
} catch {
return null;
}
}
function errorMessage(error: unknown): string {
const message = error instanceof Error ? error.message : String(error);
return message.length > 1000 ? `${message.slice(0, 1000)}…` : message;
}
export interface StartJobOptions {
kind: JobKind;
slug: string;
title: string;
steps: { id: string; label: string }[];
run: (ctx: JobContext) => Promise<JobResult | void>;
}
export interface StartedJob {
jobId: string;
/** Resolves (never rejects) with the final record once the job has finished. */
done: Promise<JobRecord>;
}
/**
* Acquires the business lock (throws BusinessLockedError if busy), writes
* the initial record, and starts `run` in the background. The returned
* promise resolves as soon as the job is visible to pollers.
*/
export async function startJob(options: StartJobOptions): Promise<StartedJob> {
const id = `${Date.now().toString(36)}-${randomHex(4)}`;
await acquireLock(options.slug, { jobId: id, kind: options.kind });
const record: JobRecord = {
id,
kind: options.kind,
slug: options.slug,
title: options.title,
status: "running",
createdAt: new Date().toISOString(),
finishedAt: null,
steps: options.steps.map((s) => ({ id: s.id, label: s.label, status: "pending", startedAt: null, finishedAt: null, detail: null })),
logTail: [],
error: null,
result: null,
};
const persist = () => withMutex(`job:${id}`, () => writeJsonFileAtomic(jobPath(id), record, 0o600));
try {
await persist();
} catch (error) {
await releaseLock(options.slug, id);
throw error;
}
runningJobIds().add(id);
const findStep = (stepId: string): JobStep => {
const step = record.steps.find((s) => s.id === stepId);
if (!step) throw new Error(`Unknown job step "${stepId}".`);
return step;
};
const ctx: JobContext = {
jobId: id,
log(line) {
const clean = line.replace(/\s+/g, " ").trim().slice(0, LOG_LINE_MAX);
if (!clean) return;
record.logTail.push(`${new Date().toISOString().slice(11, 19)} ${clean}`);
if (record.logTail.length > LOG_TAIL_LINES) record.logTail.splice(0, record.logTail.length - LOG_TAIL_LINES);
void persist().catch(() => undefined);
},
async step(stepId, fn) {
const step = findStep(stepId);
step.status = "running";
step.startedAt = new Date().toISOString();
await persist();
try {
const value = await fn();
step.status = "done";
step.finishedAt = new Date().toISOString();
await persist();
return value;
} catch (error) {
step.status = "failed";
step.finishedAt = new Date().toISOString();
step.detail = errorMessage(error);
await persist();
throw error;
}
},
skip(stepId, detail) {
const step = findStep(stepId);
step.status = "skipped";
step.detail = detail ?? null;
void persist().catch(() => undefined);
},
setResult(result) {
record.result = result;
},
};
const heartbeat = setInterval(() => {
void touchLock(options.slug, id).catch(() => undefined);
}, LOCK_HEARTBEAT_MS);
heartbeat.unref?.();
const done = (async (): Promise<JobRecord> => {
try {
const result = await options.run(ctx);
if (result) record.result = result;
record.status = "succeeded";
} catch (error) {
record.status = "failed";
record.error = errorMessage(error);
for (const step of record.steps) if (step.status === "pending") step.status = "skipped";
} finally {
clearInterval(heartbeat);
record.finishedAt = new Date().toISOString();
try {
await persist();
} catch {
// The in-memory record is still returned to in-process callers.
}
runningJobIds().delete(id);
await releaseLock(options.slug, id).catch(() => undefined);
}
return record;
})();
return { jobId: id, done };
}
/**
* Boot-time recovery: any job still "running" on disk when this process
* starts was killed with the previous process. Mark it failed so the UI
* stops spinning, and release its lock so the business isn't stuck busy
* until the stale-lock timeout.
*/
export async function recoverInterruptedJobs(): Promise<number> {
let files: string[];
try {
files = await readdir(jobsDir());
} catch {
return 0;
}
const interruptedBySlug = new Map<string, Set<string>>();
let recovered = 0;
for (const file of files) {
if (!file.endsWith(".json")) continue;
const id = file.slice(0, -".json".length);
if (runningJobIds().has(id)) continue;
const job = await readJob(id);
if (!job || job.status !== "running") continue;
job.status = "failed";
job.error = "Interrupted — the Manager restarted while this job was running. Check the business's state, then retry.";
job.finishedAt = new Date().toISOString();
for (const step of job.steps) {
if (step.status === "running") step.status = "failed";
else if (step.status === "pending") step.status = "skipped";
}
await writeJsonFileAtomic(jobPath(id), job, 0o600);
const set = interruptedBySlug.get(job.slug) ?? new Set<string>();
set.add(id);
interruptedBySlug.set(job.slug, set);
recovered += 1;
}
for (const [slug, ids] of interruptedBySlug) await releaseLockIfOwnedBy(slug, ids);
return recovered;
}
/** Deletes finished job records older than `maxAgeMs` (default 14 days). */
export async function pruneOldJobs(maxAgeMs = 14 * 24 * 60 * 60 * 1000, now: Date = new Date()): Promise<number> {
let files: string[];
try {
files = await readdir(jobsDir());
} catch {
return 0;
}
let removed = 0;
for (const file of files) {
if (!file.endsWith(".json")) continue;
const id = file.slice(0, -".json".length);
const job = await readJob(id);
if (!job || job.status === "running" || !job.finishedAt) continue;
if (now.getTime() - Date.parse(job.finishedAt) > maxAgeMs) {
await rm(jobPath(id), { force: true });
removed += 1;
}
}
return removed;
}
+20
View File
@@ -0,0 +1,20 @@
import { mkdir, readFile, rename, writeFile } from "node:fs/promises";
import path from "node:path";
import { randomHex } from "./random";
/** Missing/corrupt file → the fallback, never a throw: these files are operational state, and a bad one must not take the Manager down. */
export async function readJsonFile<T>(filePath: string, fallback: T): Promise<T> {
try {
return JSON.parse(await readFile(filePath, "utf8")) as T;
} catch {
return fallback;
}
}
/** Write-then-rename so a reader (or a crash mid-write) never sees a half-written file. */
export async function writeJsonFileAtomic(filePath: string, data: unknown, mode = 0o600): Promise<void> {
await mkdir(path.dirname(filePath), { recursive: true });
const tmpPath = `${filePath}.${randomHex(4)}.tmp`;
await writeFile(tmpPath, JSON.stringify(data, null, 2), { encoding: "utf8", mode });
await rename(tmpPath, filePath);
}
+121
View File
@@ -0,0 +1,121 @@
import { mkdir, open, readFile, rm } from "node:fs/promises";
import path from "node:path";
import { writeJsonFileAtomic } from "./json-file";
import { locksDir } from "./manager-paths";
import { withMutex } from "./singleton";
/**
* Per-business lock: at most one long-running operation (update, backup,
* restore, delete) per business at a time. A file under
* STACKS_DIR/manager/locks/<slug>.json, created with the exclusive-create
* flag so two acquirers can never both win.
*
* Stale-lock rule: the holder refreshes `heartbeatAt` every few seconds
* while it runs (see jobs.ts). A lock whose heartbeat is older than
* STALE_LOCK_MS belongs to a process that died (the Manager was
* restarted mid-job) and may be taken over. On boot the Manager also
* fails any job still marked "running" and releases its lock outright
* (jobs.ts → recoverInterruptedJobs).
*/
export const STALE_LOCK_MS = 2 * 60 * 1000;
export const LOCK_HEARTBEAT_MS = 15 * 1000;
export interface BusinessLock {
slug: string;
jobId: string;
kind: string;
acquiredAt: string;
heartbeatAt: string;
}
export class BusinessLockedError extends Error {
constructor(readonly lock: BusinessLock) {
super(`"${lock.slug}" is busy — a ${lock.kind} job (${lock.jobId}) is already running for it. Wait for it to finish.`);
this.name = "BusinessLockedError";
}
}
function lockPath(slug: string): string {
return path.join(locksDir(), `${slug}.json`);
}
export async function readLock(slug: string): Promise<BusinessLock | null> {
try {
return JSON.parse(await readFile(lockPath(slug), "utf8")) as BusinessLock;
} catch {
return null;
}
}
export function isLockStale(lock: BusinessLock, now: Date = new Date()): boolean {
const heartbeat = Date.parse(lock.heartbeatAt);
if (Number.isNaN(heartbeat)) return true;
return now.getTime() - heartbeat > STALE_LOCK_MS;
}
/** The live (non-stale) lock for a business, or null. */
export async function getActiveLock(slug: string, now: Date = new Date()): Promise<BusinessLock | null> {
const lock = await readLock(slug);
return lock && !isLockStale(lock, now) ? lock : null;
}
/** Throws BusinessLockedError if a live job holds the business — for quick actions (start/stop/restart) that must not run mid-update or mid-restore. */
export async function assertNotLocked(slug: string): Promise<void> {
const lock = await getActiveLock(slug);
if (lock) throw new BusinessLockedError(lock);
}
export function acquireLock(slug: string, owner: { jobId: string; kind: string }, now: Date = new Date()): Promise<BusinessLock> {
// The mutex covers this process (separate bundles share it via globalThis);
// the exclusive-create flag below covers everything else.
return withMutex("business-locks", async () => {
await mkdir(locksDir(), { recursive: true });
const existing = await readLock(slug);
if (existing && !isLockStale(existing, now)) throw new BusinessLockedError(existing);
if (existing) await rm(lockPath(slug), { force: true });
const lock: BusinessLock = { slug, jobId: owner.jobId, kind: owner.kind, acquiredAt: now.toISOString(), heartbeatAt: now.toISOString() };
let handle;
try {
handle = await open(lockPath(slug), "wx", 0o600);
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "EEXIST") {
const winner = await readLock(slug);
if (winner) throw new BusinessLockedError(winner);
}
throw error;
}
try {
await handle.writeFile(JSON.stringify(lock, null, 2), "utf8");
} finally {
await handle.close();
}
return lock;
});
}
/** Refreshes the heartbeat — only if this job still owns the lock. */
export function touchLock(slug: string, jobId: string, now: Date = new Date()): Promise<void> {
return withMutex("business-locks", async () => {
const lock = await readLock(slug);
if (!lock || lock.jobId !== jobId) return;
await writeJsonFileAtomic(lockPath(slug), { ...lock, heartbeatAt: now.toISOString() });
});
}
/** Only the owner releases its lock — a job that was declared stale and taken over must not delete its successor's lock. */
export function releaseLock(slug: string, jobId: string): Promise<void> {
return withMutex("business-locks", async () => {
const lock = await readLock(slug);
if (lock && lock.jobId === jobId) await rm(lockPath(slug), { force: true });
});
}
/** Boot-time cleanup helper: drops the lock if (and only if) it belongs to one of the given interrupted jobs. */
export function releaseLockIfOwnedBy(slug: string, jobIds: Set<string>): Promise<void> {
return withMutex("business-locks", async () => {
const lock = await readLock(slug);
if (lock && jobIds.has(lock.jobId)) await rm(lockPath(slug), { force: true });
});
}
+45
View File
@@ -0,0 +1,45 @@
import path from "node:path";
import { stacksRoot } from "./stacks";
/** Manager-owned state lives under STACKS_DIR/manager (already used by global-settings.json), so it survives container rebuilds with no extra volume. */
export function managerDataDir(): string {
return path.join(stacksRoot(), "manager");
}
export function jobsDir(): string {
return path.join(managerDataDir(), "jobs");
}
export function locksDir(): string {
return path.join(managerDataDir(), "locks");
}
export function schedulerStatePath(): string {
return path.join(managerDataDir(), "scheduler-state.json");
}
export function healthStatePath(): string {
return path.join(managerDataDir(), "health-state.json");
}
export function backupSettingsPath(): string {
return path.join(managerDataDir(), "backup-settings.json");
}
/** STACKS_DIR/backups/<slug>/ — deliberately outside the business's own stack directory, so deleting a business never deletes its backups. */
export function backupsRoot(): string {
return path.join(stacksRoot(), "backups");
}
export function backupDir(slug: string): string {
return path.join(backupsRoot(), slug);
}
/** Uploaded .7z files waiting to be restored; swept after a day. */
export function uploadsDir(): string {
return path.join(backupsRoot(), "_uploads");
}
export function stagingRoot(): string {
return path.join(managerDataDir(), "tmp");
}
+160
View File
@@ -0,0 +1,160 @@
import { performBackup, BACKUP_STEPS } from "./backup";
import type { BackupKind } from "./backup-manifest";
import { recordBackupOutcome } from "./backup-settings";
import { composeUp } from "./docker";
import { readBackupPassphrase, readOperatorSettings } from "./global-settings";
import { setIntentionallyStopped } from "./health-watcher";
import { waitForHealth } from "./health";
import { startJob, type JobContext, type StartedJob } from "./jobs";
import { sendOperatorEmail } from "./operator-email";
import { deleteBusiness } from "./provisioning";
import { performRestore, readArchiveManifest, RESTORE_STEPS, resolveArchivePath, type RestoreSource } from "./restore";
import { isValidSlug, stackDir, stackExists } from "./stacks";
/**
* The long-running operations, each as a background job holding the
* per-business lock (jobs.ts): Update, Backup, Restore, Delete. Server
* actions and the scheduler start them; the UI polls the job record.
* Everything that can fail partway reports it through the job — nothing
* here is awaited by a request.
*/
async function requirePassphrase(): Promise<string> {
const passphrase = await readBackupPassphrase();
if (!passphrase) throw new Error("No backup passphrase exists yet — open a business's Backups page first so one is generated.");
return passphrase;
}
async function alertBackupFailure(slug: string, message: string): Promise<void> {
try {
const settings = await readOperatorSettings();
if (!settings.alertOnBackupFailure || !settings.alertEmail) return;
await sendOperatorEmail(`Backup FAILED for ${slug}`, `The backup of "${slug}" failed.\n\nReason: ${message}\n\nCheck the Manager's Backups page for that business.\n`);
} catch {
// Alerting is best-effort; the failure is already in the job record and backup status.
}
}
export async function startBackupJob(slug: string, kind: BackupKind = "regular"): Promise<StartedJob> {
if (!isValidSlug(slug) || !(await stackExists(slug))) throw new Error(`No such business: ${slug}`);
await requirePassphrase();
return startJob({
kind: "backup",
slug,
title: `Back up ${slug}`,
steps: [...BACKUP_STEPS],
run: async (ctx) => {
try {
const result = await performBackup(slug, kind, ctx);
if (kind === "regular") await recordBackupOutcome(slug, { ok: true, message: `Saved ${result.fileName}` });
return { fileName: result.fileName, sizeBytes: result.sizeBytes, pruned: result.pruned.length };
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
if (kind === "regular") {
await recordBackupOutcome(slug, { ok: false, message });
await alertBackupFailure(slug, message);
}
throw error;
}
},
});
}
/** Recreates the container against the shared image tag; optionally takes a safety backup first. Rebuilding the image stays a manual step. */
export async function startUpdateJob(slug: string, backupFirst: boolean): Promise<StartedJob> {
if (!isValidSlug(slug) || !(await stackExists(slug))) throw new Error(`No such business: ${slug}`);
if (backupFirst) await requirePassphrase();
return startJob({
kind: "update",
slug,
title: `Update ${slug}`,
steps: [
...(backupFirst ? [{ id: "backup", label: "Back up first" }] : []),
{ id: "recreate", label: "Recreate the container" },
{ id: "health", label: "Wait for /api/health" },
{ id: "done", label: "Done" },
],
run: async (ctx) => {
if (backupFirst) {
await ctx.step("backup", async () => {
const result = await performBackup(slug, "safety", { log: ctx.log });
ctx.log(`Pre-update backup: ${result.fileName}`);
});
}
await ctx.step("recreate", async () => {
ctx.log("docker compose up -d --force-recreate");
await composeUp(stackDir(slug), { forceRecreate: true });
});
await ctx.step("health", async () => {
let lastSeen = "";
await waitForHealth(slug, {
onAttempt: (s) => {
const seen = `HTTP ${s.httpStatus ?? "no response"}`;
if (seen !== lastSeen) ctx.log(`health: ${seen}`);
lastSeen = seen;
},
});
});
await ctx.step("done", async () => {
ctx.log("The container is running the current shared image and answering /api/health.");
});
},
});
}
/** Takes an optional final backup, then removes the business. If the backup fails the business is left untouched. */
export async function startDeleteJob(slug: string, finalBackup: boolean): Promise<StartedJob> {
if (!isValidSlug(slug) || !(await stackExists(slug))) throw new Error(`No such business: ${slug}`);
if (finalBackup) await requirePassphrase();
return startJob({
kind: "delete",
slug,
title: `Delete ${slug}`,
steps: [
{ id: "final-backup", label: "Take a final backup" },
{ id: "delete", label: "Remove the business" },
],
run: async (ctx: JobContext) => {
let finalName: string | null = null;
if (finalBackup) {
finalName = await ctx.step("final-backup", async () => (await performBackup(slug, "final", { log: ctx.log })).fileName);
} else {
ctx.skip("final-backup", "Skipped by the operator.");
}
const result = await ctx.step("delete", async () => deleteBusiness(slug));
await setIntentionallyStopped(slug, false);
return { finalBackup: finalName, warnings: result.warnings.join("; ") || null };
},
});
}
export interface RestoreJobInput {
source: RestoreSource;
/** Blank → the stored passphrase. */
passphrase: string;
replace: boolean;
}
export async function startRestoreJob(input: RestoreJobInput): Promise<StartedJob> {
const passphrase = input.passphrase.trim() || (await readBackupPassphrase()) || "";
if (!passphrase) throw new Error("Enter the backup passphrase.");
const archivePath = resolveArchivePath(input.source);
// The lock is per slug and the slug lives in the archive, so read the manifest first (fast: only the encrypted header is opened).
const manifest = await readArchiveManifest(archivePath, passphrase);
return startJob({
kind: "restore",
slug: manifest.slug,
title: `Restore ${manifest.slug}`,
steps: [...RESTORE_STEPS],
run: async (ctx) => {
const result = await performRestore({ archivePath, passphrase, replace: input.replace, ctx });
await setIntentionallyStopped(result.slug, false);
return { slug: result.slug, domain: result.domain, safetyBackup: result.safetyBackup };
},
});
}
+28
View File
@@ -0,0 +1,28 @@
import { isEmailConfigComplete, readOperatorSettings, resolveEmailSettings } from "./global-settings";
import { sendPlainSmtpEmail } from "./test-email";
export interface OperatorEmailResult {
ok: boolean;
message: string;
}
/**
* Emails the operator alert address using the Manager's existing SMTP
* settings (Settings > Email) — the same pattern as the Settings page's
* test email. Never throws; a missing address or incomplete SMTP config
* is reported as `ok: false` so the caller can retry later.
*/
export async function sendOperatorEmail(subject: string, text: string): Promise<OperatorEmailResult> {
const operator = await readOperatorSettings();
if (!operator.alertEmail) return { ok: false, message: "No operator alert email is configured." };
let smtp;
try {
smtp = await resolveEmailSettings();
} catch {
return { ok: false, message: "Couldn't read the SMTP settings (is MANAGER_SETTINGS_ENCRYPTION_KEY set?)." };
}
if (!smtp || !isEmailConfigComplete(smtp)) return { ok: false, message: "SMTP isn't fully configured in Settings > Email." };
return sendPlainSmtpEmail({ ...smtp, recipient: operator.alertEmail, subject: `[NextChair] ${subject}`, text });
}
+178
View File
@@ -0,0 +1,178 @@
import { spawn, type ChildProcess } from "node:child_process";
import { createReadStream, createWriteStream } from "node:fs";
import path from "node:path";
import { pipeline } from "node:stream/promises";
/**
* Streaming subprocess helpers. Everything big (database dumps, media
* tars, archives) flows through spawn + pipes straight to/from disk —
* never execFile, whose maxBuffer would kill a large dump mid-stream.
*
* Commands are always `{ command, args[] }` (never a shell string), and
* secrets travel in `env` (e.g. MYSQL_PWD) or the 7zz passphrase arg —
* never in anything this module logs: errors carry only the command's
* basename and its stderr tail.
*/
export interface CommandSpec {
command: string;
args: string[];
/** Merged over process.env. */
env?: Record<string, string>;
cwd?: string;
}
export class CommandError extends Error {
constructor(
message: string,
readonly exitCode: number | null,
readonly stderrTail: string,
) {
super(message);
this.name = "CommandError";
}
}
const STDERR_TAIL_BYTES = 4000;
function spawnSpec(spec: CommandSpec, stdio: ("pipe" | "ignore")[]): ChildProcess {
return spawn(spec.command, spec.args, {
cwd: spec.cwd,
env: { ...process.env, ...spec.env },
stdio,
});
}
interface Completion {
done: Promise<void>;
stderrTail: () => string;
}
/** Resolves on a clean exit; rejects (with the stderr tail) on spawn failure or non-zero exit. */
function watch(child: ChildProcess, spec: CommandSpec): Completion {
let tail = "";
child.stderr?.on("data", (chunk: Buffer) => {
tail = (tail + chunk.toString("utf8")).slice(-STDERR_TAIL_BYTES);
});
const done = new Promise<void>((resolve, reject) => {
child.once("error", (error) => {
reject(new CommandError(`Couldn't run ${path.basename(spec.command)}: ${error.message}`, null, tail));
});
child.once("close", (code, signal) => {
if (code === 0) resolve();
else {
const reason = signal ? `was killed (${signal})` : `exited with code ${code}`;
reject(new CommandError(`${path.basename(spec.command)} ${reason}${tail.trim() ? `: ${tail.trim()}` : ""}`, code, tail));
}
});
});
return { done, stderrTail: () => tail };
}
/** Runs a command and returns its (size-capped) stdout — for SMALL output only (a version string, a one-row query). */
export async function runCapture(
spec: CommandSpec,
options: { maxBytes?: number; timeoutMs?: number } = {},
): Promise<{ stdout: string; stderr: string }> {
const maxBytes = options.maxBytes ?? 1024 * 1024;
const child = spawnSpec(spec, ["ignore", "pipe", "pipe"]);
const { done, stderrTail } = watch(child, spec);
const chunks: Buffer[] = [];
let size = 0;
child.stdout?.on("data", (chunk: Buffer) => {
size += chunk.length;
if (size > maxBytes) {
child.kill("SIGKILL");
return;
}
chunks.push(chunk);
});
const timer = options.timeoutMs ? setTimeout(() => child.kill("SIGKILL"), options.timeoutMs) : null;
try {
await done;
} catch (error) {
// We killed it for exceeding the cap — say that, not the resulting SIGKILL.
if (size > maxBytes) throw new CommandError(`${path.basename(spec.command)} produced more than ${maxBytes} bytes of output.`, null, "");
throw error;
} finally {
if (timer) clearTimeout(timer);
}
if (size > maxBytes) throw new CommandError(`${path.basename(spec.command)} produced more than ${maxBytes} bytes of output.`, null, "");
return { stdout: Buffer.concat(chunks).toString("utf8"), stderr: stderrTail() };
}
/** Runs a command that needs no output captured at all (e.g. `7zz a`/`7zz t`) — only the stderr tail is kept for error reporting. */
export async function runQuiet(spec: CommandSpec): Promise<void> {
const child = spawnSpec(spec, ["ignore", "ignore", "pipe"]);
await watch(child, spec).done;
}
/** stdout → file, streamed. The file is written with 0600 permissions. */
export async function runToFile(spec: CommandSpec, filePath: string): Promise<void> {
const child = spawnSpec(spec, ["ignore", "pipe", "pipe"]);
const { done } = watch(child, spec);
// Attach the failure handler before awaiting the pipeline so a fast
// non-zero exit can't surface as an unhandled rejection.
const settled = done.then(
() => null,
(error: unknown) => error,
);
try {
await pipeline(child.stdout!, createWriteStream(filePath, { mode: 0o600 }));
} catch (error) {
child.kill("SIGKILL");
throw (await settled) ?? error;
}
const failure = await settled;
if (failure) throw failure;
}
/** file → stdin, streamed. */
export async function runFromFile(spec: CommandSpec, filePath: string): Promise<void> {
const child = spawnSpec(spec, ["pipe", "ignore", "pipe"]);
const { done } = watch(child, spec);
const settled = done.then(
() => null,
(error: unknown) => error,
);
child.stdin?.on("error", () => undefined); // EPIPE when the command exits early — its exit code tells the real story
try {
await pipeline(createReadStream(filePath), child.stdin!);
} catch (error) {
child.kill("SIGKILL");
throw (await settled) ?? error;
}
const failure = await settled;
if (failure) throw failure;
}
/** source.stdout → sink.stdin (e.g. `7zz x -so … | docker exec -i … mysql`). Fails if either side fails; the failing side's stderr is reported. */
export async function runPipeline(source: CommandSpec, sink: CommandSpec): Promise<void> {
const producer = spawnSpec(source, ["ignore", "pipe", "pipe"]);
const consumer = spawnSpec(sink, ["pipe", "ignore", "pipe"]);
const producerWatch = watch(producer, source);
const consumerWatch = watch(consumer, sink);
const producerResult = producerWatch.done.then(
() => null,
(error: unknown) => error,
);
const consumerResult = consumerWatch.done.then(
() => null,
(error: unknown) => error,
);
consumer.stdin?.on("error", () => undefined);
producer.stdout?.pipe(consumer.stdin!);
const [producerError, consumerError] = await Promise.all([producerResult, consumerResult]);
// If the producer died, the consumer saw a truncated stream and may
// have "succeeded" on it — the producer's failure is the root cause.
if (producerError || consumerError) {
producer.kill("SIGKILL");
consumer.kill("SIGKILL");
// When both fail, either could be the root cause (a dead consumer
// gives the producer an EPIPE; a dead producer hands the consumer a
// truncated stream) — report both rather than guess.
const messages = [producerError, consumerError].filter(Boolean).map((e) => (e instanceof Error ? e.message : String(e)));
throw new CommandError(messages.join(" | "), null, "");
}
}
+17 -4
View File
@@ -36,7 +36,7 @@ function assertValidDomain(domain: string): void {
}
}
async function assertDomainAvailable(domain: string, excludeSlug?: string): Promise<void> {
export async function assertDomainAvailable(domain: string, excludeSlug?: string): Promise<void> {
const existing = await listBusinessStacks();
const collision = existing.find((s) => s.slug !== excludeSlug && s.domain === domain);
if (collision) {
@@ -44,7 +44,7 @@ async function assertDomainAvailable(domain: string, excludeSlug?: string): Prom
}
}
async function writeCaddySiteAndReload(slug: string, domain: string): Promise<void> {
export async function writeCaddySiteAndReload(slug: string, domain: string): Promise<void> {
const sitesDir = path.join(stacksRoot(), "infra", "sites");
await mkdir(sitesDir, { recursive: true });
await writeFile(path.join(sitesDir, `${slug}.caddy`), generateCaddySite(slug, domain), "utf8");
@@ -73,7 +73,7 @@ async function connectAsRoot(): Promise<Connection> {
});
}
async function createDatabaseAndUser(databaseName: string, databaseUser: string, databasePassword: string): Promise<void> {
export async function createDatabaseAndUser(databaseName: string, databaseUser: string, databasePassword: string): Promise<void> {
if (!SAFE_DB_IDENTIFIER.test(databaseName) || !SAFE_DB_IDENTIFIER.test(databaseUser)) {
throw new Error("Refusing to create a database/user from an unsafe identifier.");
}
@@ -96,7 +96,7 @@ async function createDatabaseAndUser(databaseName: string, databaseUser: string,
}
/** Irreversible — only ever called from deleteBusiness, itself gated behind a type-the-slug confirmation in the UI. */
async function dropDatabaseAndUser(databaseName: string, databaseUser: string): Promise<void> {
export async function dropDatabaseAndUser(databaseName: string, databaseUser: string): Promise<void> {
if (!SAFE_DB_IDENTIFIER.test(databaseName) || !SAFE_DB_IDENTIFIER.test(databaseUser)) {
throw new Error("Refusing to drop a database/user from an unsafe identifier.");
}
@@ -110,6 +110,19 @@ async function dropDatabaseAndUser(databaseName: string, databaseUser: string):
}
}
/**
* Restore path: makes the database + user exist, empty, with the password
* that is already written in the (backed-up) .env — dropping whatever was
* there first, since a replacing restore starts from a clean database.
* Only the explicit-password overload of createDatabaseAndUser is used
* here, never a freshly generated one: the restored .env must keep
* working as-is.
*/
export async function recreateDatabaseAndUser(databaseName: string, databaseUser: string, databasePassword: string): Promise<void> {
await dropDatabaseAndUser(databaseName, databaseUser);
await createDatabaseAndUser(databaseName, databaseUser, databasePassword);
}
export interface ProvisionBusinessInput {
name: string;
slug: string;
+209
View File
@@ -0,0 +1,209 @@
import { mkdir, writeFile } from "node:fs/promises";
import path from "node:path";
import { performBackup } from "./backup";
import { BACKUP_MEMBERS, checkMigrationCompatibility, parseManifest, type BackupManifest } from "./backup-manifest";
import { backupFilePath } from "./backup-store";
import { generateComposeYaml } from "./compose-template";
import { composeCreate, composeDown, composeUp, listImageMigrations } from "./docker";
import { mediaRestoreCommand, mysqlImportCommand } from "./docker-commands";
import { readEnvValue } from "./env-file";
import { waitForHealth } from "./health";
import { uploadsDir } from "./manager-paths";
import { assertDomainAvailable, recreateDatabaseAndUser, writeCaddySiteAndReload } from "./provisioning";
import { runPipeline } from "./process";
import { sevenZipExtractToStdoutCommand, sevenZipReadMember, sevenZipTest } from "./sevenzip";
import { stackDir, stackExists } from "./stacks";
/**
* Restore a backup archive as a business — onto an existing business
* (replace) or onto a fresh server where it never existed. The step order
* is the contract (and is asserted in tests):
*
* verify → check → safety backup (replace only) → stop → recreate DB
* → import dump → write config + Caddy → restore media → start → health
*
* The database password comes from the backed-up .env (so that .env keeps
* working untouched); the dump already contains the "installed" row, so
* the setup wizard doesn't appear — the final step insists on
* `installed: true`. Like performBackup this does not take the business
* lock; the restore job holds it.
*/
export const RESTORE_STEPS = [
{ id: "verify", label: "Verify the archive and passphrase" },
{ id: "check", label: "Check for conflicts and schema version" },
{ id: "safety", label: "Take a safety backup of the current business" },
{ id: "stop", label: "Stop the current stack" },
{ id: "database", label: "Recreate the database and user" },
{ id: "import", label: "Import the database dump" },
{ id: "config", label: "Write configuration and the Caddy site" },
{ id: "media", label: "Restore media files" },
{ id: "start", label: "Start the stack" },
{ id: "health", label: "Wait for the app to report installed" },
] as const;
export type RestoreSource = { kind: "backup"; slug: string; fileName: string } | { kind: "upload"; uploadId: string };
const UPLOAD_ID_PATTERN = /^[a-f0-9]{16}$/;
export function resolveArchivePath(source: RestoreSource): string {
if (source.kind === "backup") return backupFilePath(source.slug, source.fileName);
if (!UPLOAD_ID_PATTERN.test(source.uploadId)) throw new Error("Invalid upload reference.");
return path.join(uploadsDir(), `${source.uploadId}.7z`);
}
export interface RestorePlan {
manifest: BackupManifest;
/** A stack with this slug already exists — restoring needs explicit "replace". */
slugExists: boolean;
/** Another business already uses the backup's domain. */
domainConflict: string | null;
schema: { ok: true } | { ok: false; reason: string };
}
/** Fast: opens only the (encrypted) header and reads manifest.json. A wrong passphrase fails here. Does not verify the payload. */
export async function readArchiveManifest(archivePath: string, passphrase: string): Promise<BackupManifest> {
return parseManifest(await sevenZipReadMember(archivePath, passphrase, BACKUP_MEMBERS.manifest));
}
/** Full verification (`7zz t` over the whole archive) plus the manifest. Throws on a corrupt archive or wrong passphrase. */
export async function inspectArchive(archivePath: string, passphrase: string): Promise<BackupManifest> {
await sevenZipTest(archivePath, passphrase);
return readArchiveManifest(archivePath, passphrase);
}
/** Read-only conflict/version report shown on the Restore page before the operator confirms. */
export async function planRestore(manifest: BackupManifest): Promise<RestorePlan> {
const slugExists = await stackExists(manifest.slug);
let domainConflict: string | null = null;
try {
await assertDomainAvailable(manifest.domain, manifest.slug);
} catch (error) {
domainConflict = error instanceof Error ? error.message : "Domain conflict.";
}
let schema: RestorePlan["schema"];
try {
schema = checkMigrationCompatibility(manifest.lastMigration, await listImageMigrations());
} catch (error) {
schema = { ok: false, reason: `Couldn't read the app image's migrations: ${error instanceof Error ? error.message : String(error)}` };
}
return { manifest, slugExists, domainConflict, schema };
}
export interface RestoreContext {
log(line: string): void;
step<T>(stepId: string, fn: () => Promise<T>): Promise<T>;
skip(stepId: string, detail?: string): void;
}
export interface RestoreResult {
slug: string;
domain: string;
safetyBackup: string | null;
}
export async function performRestore(params: {
archivePath: string;
passphrase: string;
/** Operator confirmed overwriting an existing business with the same slug. */
replace: boolean;
ctx: RestoreContext;
}): Promise<RestoreResult> {
const { archivePath, passphrase, replace, ctx } = params;
const { manifest, envContent } = await ctx.step("verify", async () => {
ctx.log("Testing the archive (7zz t)…");
const parsed = await inspectArchive(archivePath, passphrase);
const env = await sevenZipReadMember(archivePath, passphrase, BACKUP_MEMBERS.env);
ctx.log(`Backup of "${parsed.slug}" (${parsed.domain}) taken ${parsed.createdAt}.`);
// The .env inside the archive must agree with the manifest — the
// password for the new database user comes from it.
const envDb = readEnvValue(env, "DATABASE_NAME");
const envUser = readEnvValue(env, "DATABASE_USER");
const envPassword = readEnvValue(env, "DATABASE_PASSWORD");
if (envDb !== parsed.databaseName || envUser !== parsed.databaseUser || !envPassword) {
throw new Error("The archive's .env doesn't match its manifest (database name/user/password) — refusing to restore it.");
}
return { manifest: parsed, envContent: env };
});
const slug = manifest.slug;
const dir = stackDir(slug);
const exists = await stackExists(slug);
await ctx.step("check", async () => {
if (exists && !replace) {
throw new Error(`A business named "${slug}" already exists. Confirm "replace" to overwrite it (a safety backup is taken first).`);
}
await assertDomainAvailable(manifest.domain, slug);
const compatibility = checkMigrationCompatibility(manifest.lastMigration, await listImageMigrations());
if (!compatibility.ok) throw new Error(compatibility.reason);
ctx.log("No conflicts; the backup's schema is compatible with the running app image.");
});
let safetyBackup: string | null = null;
if (exists) {
safetyBackup = await ctx.step("safety", async () => {
ctx.log(`"${slug}" exists — taking a safety backup before replacing it…`);
const result = await performBackup(slug, "safety", { log: ctx.log });
ctx.log(`Safety backup: ${result.fileName}`);
return result.fileName;
});
} else {
ctx.skip("safety", "No existing business to protect.");
}
if (exists) {
await ctx.step("stop", async () => {
// -v: the old media volume goes too, so files deleted since the backup don't linger.
await composeDown(dir, { removeVolumes: true });
});
} else {
ctx.skip("stop", "Nothing running under this slug.");
}
await ctx.step("database", async () => {
await recreateDatabaseAndUser(manifest.databaseName, manifest.databaseUser, readEnvValue(envContent, "DATABASE_PASSWORD")!);
});
await ctx.step("import", async () => {
ctx.log("Importing db.sql (streamed from the archive)…");
await runPipeline(sevenZipExtractToStdoutCommand(archivePath, passphrase, BACKUP_MEMBERS.database), mysqlImportCommand(manifest.databaseName));
});
await ctx.step("config", async () => {
await mkdir(dir, { recursive: true });
await writeFile(path.join(dir, ".env"), envContent, { encoding: "utf8", mode: 0o600 });
await writeFile(path.join(dir, "compose.yaml"), generateComposeYaml(slug), "utf8");
await writeCaddySiteAndReload(slug, manifest.domain);
ctx.log("Wrote .env and compose.yaml, and reloaded Caddy.");
});
await ctx.step("media", async () => {
// Create (don't start) the container so the named volume exists with compose's labels before it's filled.
await composeCreate(dir);
await runPipeline(sevenZipExtractToStdoutCommand(archivePath, passphrase, BACKUP_MEMBERS.media), mediaRestoreCommand(slug));
ctx.log("Media restored and handed to uid 1001.");
});
await ctx.step("start", async () => {
// The entrypoint runs `prisma migrate deploy`, so an older dump upgrades itself here.
await composeUp(dir);
});
await ctx.step("health", async () => {
// Log only when the answer changes — polling every few seconds would otherwise bury the log tail.
let lastSeen = "";
await waitForHealth(slug, {
requireInstalled: true,
onAttempt: (s) => {
const seen = `HTTP ${s.httpStatus ?? "no response"}, installed=${String(s.installed)}`;
if (seen !== lastSeen) ctx.log(`health: ${seen}`);
lastSeen = seen;
},
});
});
return { slug, domain: manifest.domain, safetyBackup };
}
+112
View File
@@ -0,0 +1,112 @@
import { readJsonFile, writeJsonFileAtomic } from "./json-file";
import { recoverInterruptedJobs } from "./jobs";
import { readOperatorSettings } from "./global-settings";
import { schedulerStatePath } from "./manager-paths";
import {
emptyState,
runTick,
type SchedulerState,
type SchedulerStateStore,
type TaskState,
} from "./scheduler";
import { buildSchedulerTasks } from "./scheduler-tasks";
import { globalSingleton } from "./singleton";
import { listStackSlugs } from "./stacks";
/**
* Starts the one scheduler interval for this Manager process. Called from
* instrumentation.ts register(); guarded by a globalThis singleton so dev
* hot-reloads (or an accidental second register()) can never run two.
*/
const TICK_MS = 5000;
class FileStateStore implements SchedulerStateStore {
private writing = false;
private dirty = false;
constructor(private state: SchedulerState) {}
get(): SchedulerState {
return this.state;
}
patch(key: string, patch: Partial<TaskState>): void {
const current = this.state.tasks[key] ?? { lastStartedAt: null, lastFinishedAt: null, lastOk: null, lastMessage: null };
this.state.tasks[key] = { ...current, ...patch };
this.dirty = true;
}
setHeartbeat(at: Date): void {
this.state.heartbeatAt = at.toISOString();
this.dirty = true;
}
retain(predicate: (key: string) => boolean): void {
for (const key of Object.keys(this.state.tasks)) {
if (!predicate(key)) {
delete this.state.tasks[key];
this.dirty = true;
}
}
}
async flush(): Promise<void> {
if (this.writing) return; // the in-flight writer loops until clean
this.writing = true;
try {
while (this.dirty) {
this.dirty = false;
await writeJsonFileAtomic(schedulerStatePath(), this.state);
}
} finally {
this.writing = false;
}
}
}
interface SchedulerHandle {
timer: ReturnType<typeof setInterval>;
store: FileStateStore;
ticking: boolean;
}
export async function startScheduler(): Promise<boolean> {
const holder = globalSingleton<{ handle: SchedulerHandle | null; starting: boolean }>("scheduler_handle", () => ({ handle: null, starting: false }));
if (holder.handle || holder.starting) return false;
holder.starting = true;
try {
const recovered = await recoverInterruptedJobs().catch(() => 0);
if (recovered > 0) console.warn(`[scheduler] marked ${recovered} interrupted job(s) as failed.`);
const loaded = await readJsonFile<SchedulerState>(schedulerStatePath(), emptyState());
const store = new FileStateStore({ ...emptyState(), ...loaded, tasks: loaded.tasks ?? {} });
const tasks = buildSchedulerTasks();
const handle: SchedulerHandle = { store, ticking: false, timer: setInterval(() => void tick(), TICK_MS) };
handle.timer.unref?.();
holder.handle = handle;
async function tick(): Promise<void> {
if (handle.ticking) return; // a previous tick is still dispatching (slow disk/list)
handle.ticking = true;
try {
// Deliberately not awaiting `done`: a long backup must not stall the next dispatch.
await runTick({
tasks,
store,
listSlugs: listStackSlugs,
config: readOperatorSettings,
now: () => new Date(),
});
} catch (error) {
console.error("[scheduler] tick failed:", error instanceof Error ? error.message : error);
} finally {
handle.ticking = false;
}
}
console.log(`[scheduler] started (${tasks.map((t) => t.id).join(", ")}).`);
setTimeout(() => void tick(), 3000).unref?.();
return true;
} finally {
holder.starting = false;
}
}
+21
View File
@@ -0,0 +1,21 @@
import { readJsonFile } from "./json-file";
import { schedulerStatePath } from "./manager-paths";
import { emptyState, type SchedulerState, type TaskState } from "./scheduler";
export interface SchedulerStatusRow {
key: string;
state: TaskState;
}
export interface SchedulerStatus {
heartbeatAt: string | null;
rows: SchedulerStatusRow[];
}
/** Read-only view of scheduler-state.json for the Settings page — failures first. */
export async function readSchedulerStatus(): Promise<SchedulerStatus> {
const state = await readJsonFile<SchedulerState>(schedulerStatePath(), emptyState());
const rows = Object.entries(state.tasks ?? {}).map(([key, taskState]) => ({ key, state: taskState }));
rows.sort((a, b) => Number(a.state.lastOk === true) - Number(b.state.lastOk === true) || a.key.localeCompare(b.key));
return { heartbeatAt: state.heartbeatAt ?? null, rows };
}
+152
View File
@@ -0,0 +1,152 @@
import { readdir, rm, stat } from "node:fs/promises";
import path from "node:path";
import { isBackupDue, readBackupSettings, recordBackupAttempt } from "./backup-settings";
import { readEnvFile, readEnvValue } from "./env-file";
import { runHealthCheck } from "./health-watcher";
import { pruneOldJobs } from "./jobs";
import { BusinessLockedError } from "./locks";
import { stagingRoot, uploadsDir } from "./manager-paths";
import { startBackupJob } from "./operations";
import type { SchedulerTask, TaskResult } from "./scheduler";
import { stackDir } from "./stacks";
/**
* The tasks the built-in scheduler runs. They replace the host crontab
* calling stacks/infra/scripts/process-cron.sh (kept documented as a
* fallback), and add health checks, scheduled backups and housekeeping.
*/
const NOTIFICATIONS_TIMEOUT_MS = 25_000;
const REVIEWS_TIMEOUT_MS = 120_000;
/** POSTs to a business's CRON_SECRET-gated endpoint over the internal Docker network — the same host Caddy proxies to. */
export async function callCronEndpoint(slug: string, endpoint: string, timeoutMs: number): Promise<TaskResult> {
const env = await readEnvFile(path.join(stackDir(slug), ".env"));
const secret = env ? readEnvValue(env, "CRON_SECRET") : null;
if (!secret) return { ok: false, message: "no CRON_SECRET in this business's .env" };
try {
const response = await fetch(`http://next-chair-${slug}:3000/api/cron/${endpoint}`, {
method: "POST",
headers: { authorization: `Bearer ${secret}` },
cache: "no-store",
signal: AbortSignal.timeout(timeoutMs),
});
return { ok: response.ok, message: `HTTP ${response.status}` };
} catch {
// Not the error text: it can contain the URL, which is fine, but "unreachable" is all an operator needs and is stable for de-duplication.
return { ok: false, message: "unreachable (stopped, restarting or timed out)" };
}
}
const notificationsTask: SchedulerTask = {
id: "notifications",
label: "Process notifications",
scope: "business",
intervalMs: (config) => config.notificationsIntervalSeconds * 1000,
run: ({ slug }) => callCronEndpoint(slug!, "process-notifications", NOTIFICATIONS_TIMEOUT_MS),
};
const reviewsTask: SchedulerTask = {
id: "reviews",
label: "Sync Google reviews (daily)",
scope: "business",
intervalMs: () => 24 * 60 * 60 * 1000,
run: ({ slug }) => callCronEndpoint(slug!, "sync-google-reviews", REVIEWS_TIMEOUT_MS),
};
const healthTask: SchedulerTask = {
id: "health",
label: "Health check",
scope: "business",
intervalMs: (config) => config.healthIntervalMinutes * 60 * 1000,
run: async ({ slug }) => {
const outcome = await runHealthCheck(slug!);
if (outcome.skipped) return { ok: true, message: `skipped: ${outcome.skipped}` };
const status = outcome.record?.status ?? "ok";
return { ok: status === "ok", message: `${status}${outcome.alertSent ? ` (alert sent: ${outcome.alertSent})` : ""}` };
},
};
const backupsTask: SchedulerTask = {
id: "backups",
label: "Scheduled backup",
scope: "business",
intervalMs: () => 60_000, // unused — isDue decides
isDue: async ({ slug, now }) => isBackupDue(await readBackupSettings(slug!), now),
run: async ({ slug, now }) => {
try {
// Record the attempt first: a failing backup retries at the next scheduled time, not on every tick.
const started = await startBackupJob(slug!, "regular");
await recordBackupAttempt(slug!, now);
const job = await started.done;
return job.status === "succeeded"
? { ok: true, message: `saved ${String(job.result?.fileName ?? "backup")}` }
: { ok: false, message: job.error ?? "backup failed" };
} catch (error) {
if (error instanceof BusinessLockedError) return { ok: true, message: "busy with another job — will retry" };
await recordBackupAttempt(slug!, now);
return { ok: false, message: error instanceof Error ? error.message : String(error) };
}
},
};
async function sweepOlderThan(dir: string, maxAgeMs: number, now: Date): Promise<number> {
let names: string[];
try {
names = await readdir(dir);
} catch {
return 0;
}
let removed = 0;
for (const name of names) {
const full = path.join(dir, name);
try {
const info = await stat(full);
if (now.getTime() - info.mtimeMs > maxAgeMs) {
await rm(full, { recursive: true, force: true });
removed += 1;
}
} catch {
// vanished
}
}
return removed;
}
const maintenanceTask: SchedulerTask = {
id: "maintenance",
label: "Housekeeping",
scope: "global",
intervalMs: () => 60 * 60 * 1000,
run: async ({ now }) => {
const jobs = await pruneOldJobs(undefined, now);
// Uploaded archives not restored within a day, and staging folders a dead backup left behind.
const uploads = await sweepOlderThan(uploadsDir(), 24 * 60 * 60 * 1000, now);
const staging = await sweepOlderThan(stagingRoot(), 24 * 60 * 60 * 1000, now);
return { ok: true, message: `removed ${jobs} old job record(s), ${uploads} stale upload(s), ${staging} stale staging folder(s)` };
},
};
/** Optional dead-man's-switch: pings an external URL (healthchecks.io, Better Stack…) so someone notices when the whole VPS or the Manager is down. */
const heartbeatTask: SchedulerTask = {
id: "heartbeat",
label: "External heartbeat ping",
scope: "global",
intervalMs: () => 5 * 60 * 1000,
run: async () => {
const url = process.env.MANAGER_HEARTBEAT_URL;
if (!url) return { ok: true, message: "not configured" };
try {
const response = await fetch(url, { cache: "no-store", signal: AbortSignal.timeout(10_000) });
return { ok: response.ok, message: `HTTP ${response.status}` };
} catch {
return { ok: false, message: "ping failed" };
}
},
};
export function buildSchedulerTasks(): SchedulerTask[] {
const tasks = [notificationsTask, reviewsTask, healthTask, backupsTask, maintenanceTask];
if (process.env.MANAGER_HEARTBEAT_URL) tasks.push(heartbeatTask);
return tasks;
}
+176
View File
@@ -0,0 +1,176 @@
import { globalSingleton } from "./singleton";
/**
* The scheduler engine: a registry of tasks, each with its own interval,
* driven by ONE periodic tick (see scheduler-start.ts). Everything
* environment-specific (which businesses exist, the clock, where state is
* kept, the config) is injected, so due-time and overlap behaviour are
* plain unit tests.
*
* Guards:
* - per-(task, business) overlap guard: a run still in flight is never
* started again, however slow the business is;
* - "last started" is persisted (scheduler-state.json), so a Manager
* restart doesn't make every once-a-day task fire again.
*/
export interface TaskResult {
ok: boolean;
message: string;
}
export interface TaskRunContext {
/** null for global tasks. */
slug: string | null;
now: Date;
}
export interface SchedulerConfig {
notificationsIntervalSeconds: number;
healthIntervalMinutes: number;
}
export interface SchedulerTask {
id: string;
label: string;
scope: "global" | "business";
intervalMs(config: SchedulerConfig): number;
/** Replaces the interval check (e.g. backups are due by wall-clock schedule, not by elapsed time). */
isDue?(ctx: TaskRunContext & { lastStartedAt: Date | null }): Promise<boolean>;
run(ctx: TaskRunContext): Promise<TaskResult>;
}
export interface TaskState {
lastStartedAt: string | null;
lastFinishedAt: string | null;
lastOk: boolean | null;
lastMessage: string | null;
}
export interface SchedulerState {
version: 1;
heartbeatAt: string | null;
tasks: Record<string, TaskState>;
}
export interface SchedulerStateStore {
get(): SchedulerState;
patch(key: string, patch: Partial<TaskState>): void;
setHeartbeat(at: Date): void;
/** Drops entries for tasks/businesses that no longer exist. */
retain(predicate: (key: string) => boolean): void;
flush(): Promise<void>;
}
export function taskKey(taskId: string, slug: string | null): string {
return slug ? `${taskId}:${slug}` : taskId;
}
export function emptyState(): SchedulerState {
return { version: 1, heartbeatAt: null, tasks: {} };
}
/** In-memory store — the engine's default in tests; scheduler-start.ts uses a file-backed one. */
export class MemoryStateStore implements SchedulerStateStore {
private state: SchedulerState;
constructor(initial: SchedulerState = emptyState()) {
this.state = initial;
}
get(): SchedulerState {
return this.state;
}
patch(key: string, patch: Partial<TaskState>): void {
const current = this.state.tasks[key] ?? { lastStartedAt: null, lastFinishedAt: null, lastOk: null, lastMessage: null };
this.state.tasks[key] = { ...current, ...patch };
}
setHeartbeat(at: Date): void {
this.state.heartbeatAt = at.toISOString();
}
retain(predicate: (key: string) => boolean): void {
for (const key of Object.keys(this.state.tasks)) if (!predicate(key)) delete this.state.tasks[key];
}
async flush(): Promise<void> {}
}
export interface TickDeps {
tasks: SchedulerTask[];
store: SchedulerStateStore;
listSlugs(): Promise<string[]>;
config(): Promise<SchedulerConfig>;
now(): Date;
/** Keys of runs in flight. Defaults to a process-wide set so separate bundles share it. */
inFlight?: Set<string>;
}
function defaultInFlight(): Set<string> {
return globalSingleton("scheduler_in_flight", () => new Set<string>());
}
/**
* Decides what's due and starts it. Resolves as soon as everything due has
* been STARTED; `done` settles when those runs finish. The production tick
* must not await `done` — a ten-minute backup may not stall the next tick's
* dispatch of notification crons. Tests await it.
*/
export async function runTick(deps: TickDeps): Promise<{ started: string[]; done: Promise<void> }> {
const inFlight = deps.inFlight ?? defaultInFlight();
const now = deps.now();
const config = await deps.config();
const slugs = await deps.listSlugs();
const started: string[] = [];
const runs: Promise<void>[] = [];
deps.store.setHeartbeat(now);
deps.store.retain((key) => {
const [taskId, slug] = key.split(":");
if (!deps.tasks.some((t) => t.id === taskId)) return false;
return !slug || slugs.includes(slug);
});
for (const task of deps.tasks) {
const targets: (string | null)[] = task.scope === "global" ? [null] : slugs;
for (const slug of targets) {
const key = taskKey(task.id, slug);
if (inFlight.has(key)) continue;
const lastRaw = deps.store.get().tasks[key]?.lastStartedAt ?? null;
const lastStartedAt = lastRaw ? new Date(lastRaw) : null;
const ctx: TaskRunContext = { slug, now };
let due: boolean;
try {
due = task.isDue
? await task.isDue({ ...ctx, lastStartedAt })
: lastStartedAt === null || now.getTime() - lastStartedAt.getTime() >= task.intervalMs(config);
} catch {
due = false;
}
if (!due) continue;
inFlight.add(key);
started.push(key);
deps.store.patch(key, { lastStartedAt: now.toISOString() });
runs.push(
(async () => {
let result: TaskResult;
try {
result = await task.run(ctx);
} catch (error) {
result = { ok: false, message: error instanceof Error ? error.message : String(error) };
}
deps.store.patch(key, {
lastFinishedAt: deps.now().toISOString(),
lastOk: result.ok,
lastMessage: result.message.slice(0, 300),
});
inFlight.delete(key);
await deps.store.flush().catch(() => undefined);
})(),
);
}
}
await deps.store.flush().catch(() => undefined);
return { started, done: Promise.all(runs).then(() => undefined) };
}
+56
View File
@@ -0,0 +1,56 @@
import { runCapture, runQuiet, type CommandSpec } from "./process";
/**
* 7-Zip (`7zz`, from Alpine's `7zip` package — see Dockerfile) wrappers.
* Archives are 7z with AES-256 and encrypted headers (-mhe=on), so even
* the file names inside are hidden without the passphrase.
*
* The passphrase is passed as `-p<passphrase>` (7zz has no safer
* non-interactive channel). It is generated hex/base64url by the Manager
* or typed by the operator, and — like every command here — runs as an
* argument array, never through a shell.
*/
function sevenZipBinary(): string {
return process.env.SEVENZIP_BIN ?? "7zz";
}
function assertPassphrase(passphrase: string): void {
// An empty `-p` would make 7zz prompt on a terminal that doesn't exist.
if (passphrase.length < 8) throw new Error("The backup passphrase must be at least 8 characters.");
}
export function sevenZipCreateCommand(params: { archivePath: string; cwd: string; entries: string[]; passphrase: string }): CommandSpec {
assertPassphrase(params.passphrase);
return {
command: sevenZipBinary(),
args: ["a", "-t7z", "-mx=3", "-mhe=on", `-p${params.passphrase}`, "-y", "-bd", params.archivePath, ...params.entries],
cwd: params.cwd,
};
}
export function sevenZipTestCommand(archivePath: string, passphrase: string): CommandSpec {
assertPassphrase(passphrase);
return { command: sevenZipBinary(), args: ["t", "-t7z", `-p${passphrase}`, "-y", "-bd", archivePath] };
}
/** Streams one archive member to stdout (`-so`) — for piping a dump/tar straight into docker without unpacking to disk. */
export function sevenZipExtractToStdoutCommand(archivePath: string, passphrase: string, member: string): CommandSpec {
assertPassphrase(passphrase);
return { command: sevenZipBinary(), args: ["x", "-so", "-t7z", `-p${passphrase}`, "-y", "-bd", "-bso0", "-bsp0", archivePath, member] };
}
export async function sevenZipCreate(params: Parameters<typeof sevenZipCreateCommand>[0]): Promise<void> {
await runQuiet(sevenZipCreateCommand(params));
}
/** Throws if the archive is corrupt or the passphrase is wrong. */
export async function sevenZipTest(archivePath: string, passphrase: string): Promise<void> {
await runQuiet(sevenZipTestCommand(archivePath, passphrase));
}
/** For SMALL members only (manifest.json, .env) — capped at 1 MiB. */
export async function sevenZipReadMember(archivePath: string, passphrase: string, member: string): Promise<string> {
const { stdout } = await runCapture(sevenZipExtractToStdoutCommand(archivePath, passphrase, member), { maxBytes: 1024 * 1024 });
return stdout;
}
+34
View File
@@ -0,0 +1,34 @@
/**
* Process-wide state that must be shared between Next's separately
* bundled entry points (instrumentation.ts, route handlers, server
* actions) and survive dev-server module reloads. A plain module-level
* variable is NOT enough: each of those entry points can get its own copy
* of a module, so in-memory guards (the scheduler's interval, per-task
* overlap guards, the job-lock mutex) live on `globalThis` instead.
*/
export function globalSingleton<T>(key: string, init: () => T): T {
const store = globalThis as unknown as Record<string, unknown>;
const fullKey = `__nextchair_manager_${key}`;
if (!(fullKey in store)) store[fullKey] = init();
return store[fullKey] as T;
}
/**
* Serialises async work per key within this process — used around every
* read-modify-write of a shared JSON file, so a scheduler tick and a
* server action can't interleave and lose each other's update.
*/
export function withMutex<T>(key: string, fn: () => Promise<T>): Promise<T> {
const chains = globalSingleton("mutex_chains", () => new Map<string, Promise<unknown>>());
const previous = chains.get(key) ?? Promise.resolve();
const run = previous.then(fn, fn);
// Keep the chain alive but never let one failure poison the next caller.
chains.set(
key,
run.then(
() => undefined,
() => undefined,
),
);
return run;
}
+1 -1
View File
@@ -6,7 +6,7 @@
* bundle.
*/
export const RESERVED_SLUGS = new Set(["infra", "manager"]);
export const RESERVED_SLUGS = new Set(["infra", "manager", "backups", "restore"]);
const SLUG_PATTERN = /^[a-z][a-z0-9-]{1,48}[a-z0-9]$/;
+21
View File
@@ -60,6 +60,27 @@ export async function listBusinessStacks(): Promise<BusinessStackSummary[]> {
return stacks.sort((a, b) => a.slug.localeCompare(b.slug));
}
/** Slugs of every provisioned stack — directory scan only (no `docker compose ps` per stack, unlike listBusinessStacks), cheap enough for the scheduler to call every few seconds. */
export async function listStackSlugs(): Promise<string[]> {
let entries: string[];
try {
entries = await readdir(stacksRoot());
} catch {
return [];
}
const slugs: string[] = [];
for (const slug of entries) {
if (RESERVED_SLUGS.has(slug) || slug.startsWith(".")) continue;
try {
await access(path.join(stacksRoot(), slug, "compose.yaml"));
slugs.push(slug);
} catch {
// not a stack
}
}
return slugs.sort();
}
export async function stackExists(slug: string): Promise<boolean> {
try {
await access(path.join(stackDir(slug), "compose.yaml"));
+38
View File
@@ -64,6 +64,44 @@ export async function sendTestSmtpEmail(config: TestEmailConfig): Promise<TestEm
}
}
export interface PlainEmailConfig extends Omit<TestEmailConfig, "recipient"> {
recipient: string;
subject: string;
text: string;
}
/**
* Same transport + error-categorising contract as sendTestSmtpEmail, for
* the operator alert emails (health / backup failures). Plain text only;
* never throws and never echoes raw SMTP errors.
*/
export async function sendPlainSmtpEmail(config: PlainEmailConfig): Promise<TestEmailResult> {
const transporter = nodemailer.createTransport({
host: config.host,
port: config.port,
secure: config.encryption === "tls",
ignoreTLS: config.encryption === "none",
auth: { user: config.username, pass: config.password },
connectionTimeout: 10_000,
greetingTimeout: 10_000,
socketTimeout: 20_000,
});
try {
await transporter.sendMail({
from: config.fromName ? `"${config.fromName}" <${config.fromAddress}>` : config.fromAddress,
to: config.recipient,
replyTo: config.replyTo || undefined,
subject: config.subject,
text: config.text,
});
return { ok: true, message: `Sent to ${config.recipient}.` };
} catch (error) {
return { ok: false, message: categorizeSmtpError(error) };
} finally {
transporter.close();
}
}
function categorizeSmtpError(error: unknown): string {
const code = typeof error === "object" && error !== null && "code" in error ? String((error as { code?: unknown }).code) : "";
const responseCode =
+5 -1
View File
@@ -26,5 +26,9 @@ export const config = {
// Runs on every page and API route except static assets — this app has
// no public-facing content at all, unlike the main app's proxy.ts,
// which has to carve out installation/customer-facing exceptions.
matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
//
// api/backups/upload is excluded deliberately: a proxy buffers (and
// truncates at 10 MB) every request body it sees, which would corrupt a
// large backup upload. That route verifies the session cookie itself.
matcher: ["/((?!_next/static|_next/image|favicon.ico|api/backups/upload).*)"],
};