feat: restore database backups (#23978)

* feat: ProcessRepository#createSpawnDuplexStream * test: write tests for ProcessRepository#createSpawnDuplexStream * feat: StorageRepository#createGzip,createGunzip,createPlainReadStream * feat: backups util (args, create, restore, progress) * feat: wait on maintenance operation lock on boot * chore: use backup util from backup.service.ts test: update backup.service.ts tests with new util * feat: list/delete backups (maintenance services) * chore: open api fix: missing action in cli.service.ts * chore: add missing repositories to MaintenanceModule * refactor: move logSecret into module init * feat: initialise StorageCore in maintenance mode * feat: authenticate websocket requests in maintenance mode * test: add mock for new storage fns * feat: add MaintenanceEphemeralStateRepository refactor: cache the secret in memory * test: update service worker tests * feat: add external maintenance mode status * feat: synchronised status, restore db action * test: backup restore service tests * refactor: DRY end maintenance * feat: list and delete backup routes * feat: start action on boot * fix: should set status on restore end * refactor: add maintenanceStore to hold writables * feat: sync status to web app * feat: web impl. * test: various utils for testings * test: web e2e tests * test: e2e maintenance spec * test: update cli spec * chore: e2e lint * chore: lint fixes * chore: lint fixes * feat: start restore flow route * test: update e2e tests * chore: remove neon lights on maintenance action pages * fix: use 'startRestoreFlow' on onboarding page * chore: ignore any library folder in `docker/` * fix: load status on boot * feat: upload backups * refactor: permit any .sql(.gz) to be listed/restored * feat: download backups from list * fix: permit uploading just .sql files * feat: restore just .sql files * fix: don't show backups list if logged out * feat: system integrity check in restore flow * test: not providing failed backups in API anymore * test: util should also not try to use failedBackups * fix: actually assign inputStream * test: correct test backup prep. * fix: ensure task is defined to show error * test: fix docker cp command * test: update e2e web spec to select next button * test: update e2e api tests * test: refactor timeouts * chore: remove `showDelete` from maint. settings * chore: lint * chore: lint * fix: make sure backups are correctly sorted for clean up * test: update service spec * test: adjust e2e timeout * test: increase web timeouts for ci * chore: move gitignore changes * chore: additional filename validation * refactor: better typings for integrity API * feat: higher accuracy progress tracking * chore: delay lock retry * refactor: remove old maintenance settings * refactor: clean up tailwind classes * refactor: use while loop rather than recursive calls * test: update service specs * chore: check canParse too * chore: lint * fix: logic error causing infinite loop * refactor: use <ProgressBar /> from ui library * fix: create or overwrite file * chore: i18n pass, update progress bar * fix: wrong translation string * chore: update colour variables * test: update web test for new maint. page * chore: format, fix key * test: update tests to be more linter complaint & use new routines * chore: update onClick -> onAction, title -> breadcrumbs * fix: use wrench icon in admin settings sidebar * chore: add translation strings to accordion * chore: lint * refactor: move maintenance worker init into service * refactor: `maintenanceStatus` -> `getMaintenanceStatus` refactor: `integrityCheck` -> `detectPriorInstall` chore: add `v2.4.0` version refactor: `/backups/list` -> `/backups` refactor: use sendFile in download route refactor: use separate backups permissions chore: correct descriptions refactor: permit handler that doesn't return promise for sendfile * refactor: move status impl into service refactor: add active flag to maintenance status * refactor: split into database backup controller * test: split api e2e tests and passing * fix: move end button into authed default maint page * fix: also show in restore flow * fix: import getMaintenanceStatus * test: split web e2e tests * refactor: ensure detect install is consistently named * chore: ensure admin for detect install while out of maint. * refactor: remove state repository * test: update maint. worker service spec * test: split backup service spec * refactor: rename db backup routes * refactor: instead of param, allow bulk backup deletion * test: update sdk use in e2e test * test: correct deleteBackup call * fix: correct type for serverinstall response dto * chore: validate filename for deletion * test: wip * test: backups no longer take path param * refactor: scope util to database-backups instead of backups * fix: update worker controller with new route * chore: use new admin page actions * chore: remove stray comment * test: rename outdated test * refactor: getter pattern for maintenance secret * refactor: `createSpawnDuplexStream` -> `spawnDuplexStream` * refactor: prefer `Object.assign` * refactor: remove useless try {} block * refactor: prefer `type Props` refactor: prefer arrow function * refactor: use luxon API for minutesAgo * chore: remove change to gitignore * refactor: prefer `type Props` * refactor: remove async from onMount * refactor: use luxon toRelative for relative time * refactor: duplicate logic check * chore: open api * refactor: begin moving code into web//services * refactor: don't use template string with $t * test: use dialog role to match prompt * refactor: split actions into flow/restore * test: fix action value * refactor: move more service calls into web//services * chore: should void fn return * chore: bump 2.4.0 to 2.5.0 in controller * chore: bump 2.4.0 to 2.5.0 in controller * refactor: use events for web//services * chore: open api * chore: open api * refactor: don't await returned promise * refactor: remove redundant check * refactor: add `type: command` to actions * refactor: split backup entries into own component * refactor: split restore flow into separate components * refactor(web): split BackupDelete event * chore: stylings * chore: stylings * fix: don't log query failure on first boot * feat: support pg_dumpall backups * feat: display information about each backup * chore: i18n * feat: rollback to restore point on migrations failure * feat: health check after restore * chore: format * refactor: split health check into separate function * refactor: split health into repository test: write tests covering rollbacks * fix: omit 'health' requirement from createDbBackup * test(e2e): rollback test * fix: wrap text in backup entry * fix: don't shrink context menu button * fix: correct CREATE DB syntax for postgres * test: rename backups generated by test * feat: add filesize to backup response dto * feat: restore list * feat: ui work * fix: e2e test * fix: e2e test * pr feedback * pr feedback --------- Co-authored-by: Alex <alex.tran1502@gmail.com> Co-authored-by: Jason Rasmussen <jason@rasm.me>
2026-05-18 03:10:24 +03:00 · 2026-01-20 15:22:28 +00:00
parent ca0d4b283a
commit 61a9d5cbc7
81 changed files with 5585 additions and 391 deletions
@@ -5,7 +5,7 @@ import { StorageCore } from 'src/cores/storage.core';
 import { ImmichWorker, JobStatus, StorageFolder } from 'src/enum';
 import { BackupService } from 'src/services/backup.service';
 import { systemConfigStub } from 'test/fixtures/system-config.stub';
-import { mockSpawn, newTestService, ServiceMocks } from 'test/utils';
+import { mockDuplex, mockSpawn, newTestService, ServiceMocks } from 'test/utils';
 import { describe } from 'vitest';

 describe(BackupService.name, () => {
@@ -147,6 +147,7 @@ describe(BackupService.name, () => {
    beforeEach(() => {
      mocks.storage.readdir.mockResolvedValue([]);
      mocks.process.spawn.mockReturnValue(mockSpawn(0, 'data', ''));
+      mocks.process.spawnDuplexStream.mockImplementation(() => mockDuplex('command', 0, 'data', ''));
      mocks.storage.rename.mockResolvedValue();
      mocks.storage.unlink.mockResolvedValue();
      mocks.systemMetadata.get.mockResolvedValue(systemConfigStub.backupEnabled);
@@ -165,7 +166,7 @@ describe(BackupService.name, () => {
      ({ sut, mocks } = newTestService(BackupService, { config: configMock }));

      mocks.storage.readdir.mockResolvedValue([]);
-      mocks.process.spawn.mockReturnValue(mockSpawn(0, 'data', ''));
+      mocks.process.spawnDuplexStream.mockImplementation(() => mockDuplex('command', 0, 'data', ''));
      mocks.storage.rename.mockResolvedValue();
      mocks.storage.unlink.mockResolvedValue();
      mocks.systemMetadata.get.mockResolvedValue(systemConfigStub.backupEnabled);
@@ -174,14 +175,16 @@ describe(BackupService.name, () => {

      await sut.handleBackupDatabase();

-      expect(mocks.process.spawn).toHaveBeenCalled();
-      const call = mocks.process.spawn.mock.calls[0];
+      expect(mocks.process.spawnDuplexStream).toHaveBeenCalled();
+      const call = mocks.process.spawnDuplexStream.mock.calls[0];
      const args = call[1] as string[];
-      // ['--dbname', '<url>', '--clean', '--if-exists']
-      expect(args[0]).toBe('--dbname');
-      const passedUrl = args[1];
-      expect(passedUrl).not.toContain('uselibpqcompat');
-      expect(passedUrl).toContain('sslmode=require');
+      expect(args).toMatchInlineSnapshot(`
+        [
+          "postgresql://postgres:pwd@host:5432/immich?sslmode=require",
+          "--clean",
+          "--if-exists",
+        ]
+      `);
    });

    it('should run a database backup successfully', async () => {
@@ -196,21 +199,21 @@ describe(BackupService.name, () => {
      expect(mocks.storage.rename).toHaveBeenCalled();
    });

-    it('should fail if pg_dumpall fails', async () => {
-      mocks.process.spawn.mockReturnValueOnce(mockSpawn(1, '', 'error'));
-      await expect(sut.handleBackupDatabase()).rejects.toThrow('Backup failed with code 1');
+    it('should fail if pg_dump fails', async () => {
+      mocks.process.spawnDuplexStream.mockReturnValueOnce(mockDuplex('pg_dump', 1, '', 'error'));
+      await expect(sut.handleBackupDatabase()).rejects.toThrow('pg_dump non-zero exit code (1)');
    });

    it('should not rename file if pgdump fails and gzip succeeds', async () => {
-      mocks.process.spawn.mockReturnValueOnce(mockSpawn(1, '', 'error'));
-      await expect(sut.handleBackupDatabase()).rejects.toThrow('Backup failed with code 1');
+      mocks.process.spawnDuplexStream.mockReturnValueOnce(mockDuplex('pg_dump', 1, '', 'error'));
+      await expect(sut.handleBackupDatabase()).rejects.toThrow('pg_dump non-zero exit code (1)');
      expect(mocks.storage.rename).not.toHaveBeenCalled();
    });

    it('should fail if gzip fails', async () => {
-      mocks.process.spawn.mockReturnValueOnce(mockSpawn(0, 'data', ''));
-      mocks.process.spawn.mockReturnValueOnce(mockSpawn(1, '', 'error'));
-      await expect(sut.handleBackupDatabase()).rejects.toThrow('Gzip failed with code 1');
+      mocks.process.spawnDuplexStream.mockReturnValueOnce(mockDuplex('pg_dump', 0, 'data', ''));
+      mocks.process.spawnDuplexStream.mockReturnValueOnce(mockDuplex('gzip', 1, '', 'error'));
+      await expect(sut.handleBackupDatabase()).rejects.toThrow('gzip non-zero exit code (1)');
    });

    it('should fail if write stream fails', async () => {
@@ -226,9 +229,9 @@ describe(BackupService.name, () => {
    });

    it('should ignore unlink failing and still return failed job status', async () => {
-      mocks.process.spawn.mockReturnValueOnce(mockSpawn(1, '', 'error'));
+      mocks.process.spawnDuplexStream.mockReturnValueOnce(mockDuplex('pg_dump', 1, '', 'error'));
      mocks.storage.unlink.mockRejectedValue(new Error('error'));
-      await expect(sut.handleBackupDatabase()).rejects.toThrow('Backup failed with code 1');
+      await expect(sut.handleBackupDatabase()).rejects.toThrow('pg_dump non-zero exit code (1)');
      expect(mocks.storage.unlink).toHaveBeenCalled();
    });

@@ -242,12 +245,12 @@ describe(BackupService.name, () => {
      ${'17.15.1'}                          | ${17}
      ${'18.0.0'}                           | ${18}
    `(
-      `should use pg_dumpall $expectedVersion with postgres version $postgresVersion`,
+      `should use pg_dump $expectedVersion with postgres version $postgresVersion`,
      async ({ postgresVersion, expectedVersion }) => {
        mocks.database.getPostgresVersion.mockResolvedValue(postgresVersion);
        await sut.handleBackupDatabase();
-        expect(mocks.process.spawn).toHaveBeenCalledWith(
-          `/usr/lib/postgresql/${expectedVersion}/bin/pg_dumpall`,
+        expect(mocks.process.spawnDuplexStream).toHaveBeenCalledWith(
+          `/usr/lib/postgresql/${expectedVersion}/bin/pg_dump`,
          expect.any(Array),
          expect.any(Object),
        );
@@ -1,13 +1,16 @@
 import { Injectable } from '@nestjs/common';
-import { DateTime } from 'luxon';
 import path from 'node:path';
-import semver from 'semver';
-import { serverVersion } from 'src/constants';
 import { StorageCore } from 'src/cores/storage.core';
 import { OnEvent, OnJob } from 'src/decorators';
 import { DatabaseLock, ImmichWorker, JobName, JobStatus, QueueName, StorageFolder } from 'src/enum';
 import { ArgOf } from 'src/repositories/event.repository';
 import { BaseService } from 'src/services/base.service';
+import {
+  createDatabaseBackup,
+  isFailedDatabaseBackupName,
+  isValidDatabaseRoutineBackupName,
+  UnsupportedPostgresError,
+} from 'src/utils/database-backups';
 import { handlePromiseError } from 'src/utils/misc';

@Injectable()
@@ -53,16 +56,11 @@ export class BackupService extends BaseService {

    const backupsFolder = StorageCore.getBaseFolder(StorageFolder.Backups);
    const files = await this.storageRepository.readdir(backupsFolder);
-    const failedBackups = files.filter((file) => file.match(/immich-db-backup-.*\.sql\.gz\.tmp$/));
    const backups = files
-      .filter((file) => {
-        const oldBackupStyle = file.match(/immich-db-backup-\d+\.sql\.gz$/);
-        //immich-db-backup-20250729T114018-v1.136.0-pg14.17.sql.gz
-        const newBackupStyle = file.match(/immich-db-backup-\d{8}T\d{6}-v.*-pg.*\.sql\.gz$/);
-        return oldBackupStyle || newBackupStyle;
-      })
+      .filter((filename) => isValidDatabaseRoutineBackupName(filename))
      .toSorted()
      .toReversed();
+    const failedBackups = files.filter((filename) => isFailedDatabaseBackupName(filename));

    const toDelete = backups.slice(config.keepLastAmount);
    toDelete.push(...failedBackups);
@@ -75,123 +73,27 @@ export class BackupService extends BaseService {

  @OnJob({ name: JobName.DatabaseBackup, queue: QueueName.BackupDatabase })
  async handleBackupDatabase(): Promise<JobStatus> {
-    this.logger.debug(`Database Backup Started`);
-    const { database } = this.configRepository.getEnv();
-    const config = database.config;
-
-    const isUrlConnection = config.connectionType === 'url';
-
-    let connectionUrl: string = isUrlConnection ? config.url : '';
-    if (URL.canParse(connectionUrl)) {
-      // remove known bad url parameters for pg_dumpall
-      const url = new URL(connectionUrl);
-      url.searchParams.delete('uselibpqcompat');
-      connectionUrl = url.toString();
-    }
-
-    const databaseParams = isUrlConnection
-      ? ['--dbname', connectionUrl]
-      : [
-          '--username',
-          config.username,
-          '--host',
-          config.host,
-          '--port',
-          `${config.port}`,
-          '--database',
-          config.database,
-        ];
-
-    databaseParams.push('--clean', '--if-exists');
-    const databaseVersion = await this.databaseRepository.getPostgresVersion();
-    const backupFilePath = path.join(
-      StorageCore.getBaseFolder(StorageFolder.Backups),
-      `immich-db-backup-${DateTime.now().toFormat("yyyyLLdd'T'HHmmss")}-v${serverVersion.toString()}-pg${databaseVersion.split(' ')[0]}.sql.gz.tmp`,
-    );
-    const databaseSemver = semver.coerce(databaseVersion);
-    const databaseMajorVersion = databaseSemver?.major;
-
-    if (!databaseMajorVersion || !databaseSemver || !semver.satisfies(databaseSemver, '>=14.0.0 <19.0.0')) {
-      this.logger.error(`Database Backup Failure: Unsupported PostgreSQL version: ${databaseVersion}`);
-      return JobStatus.Failed;
-    }
-
-    this.logger.log(`Database Backup Starting. Database Version: ${databaseMajorVersion}`);
-
    try {
-      await new Promise<void>((resolve, reject) => {
-        const pgdump = this.processRepository.spawn(
-          `/usr/lib/postgresql/${databaseMajorVersion}/bin/pg_dumpall`,
-          databaseParams,
-          {
-            env: {
-              PATH: process.env.PATH,
-              PGPASSWORD: isUrlConnection ? new URL(connectionUrl).password : config.password,
-            },
-          },
-        );
-
-        // NOTE: `--rsyncable` is only supported in GNU gzip
-        const gzip = this.processRepository.spawn(`gzip`, ['--rsyncable']);
-        pgdump.stdout.pipe(gzip.stdin);
-
-        const fileStream = this.storageRepository.createWriteStream(backupFilePath);
-
-        gzip.stdout.pipe(fileStream);
-
-        pgdump.on('error', (err) => {
-          this.logger.error(`Backup failed with error: ${err}`);
-          reject(err);
-        });
-
-        gzip.on('error', (err) => {
-          this.logger.error(`Gzip failed with error: ${err}`);
-          reject(err);
-        });
-
-        let pgdumpLogs = '';
-        let gzipLogs = '';
-
-        pgdump.stderr.on('data', (data) => (pgdumpLogs += data));
-        gzip.stderr.on('data', (data) => (gzipLogs += data));
-
-        pgdump.on('exit', (code) => {
-          if (code !== 0) {
-            this.logger.error(`Backup failed with code ${code}`);
-            reject(`Backup failed with code ${code}`);
-            this.logger.error(pgdumpLogs);
-            return;
-          }
-          if (pgdumpLogs) {
-            this.logger.debug(`pgdump_all logs\n${pgdumpLogs}`);
-          }
-        });
-
-        gzip.on('exit', (code) => {
-          if (code !== 0) {
-            this.logger.error(`Gzip failed with code ${code}`);
-            reject(`Gzip failed with code ${code}`);
-            this.logger.error(gzipLogs);
-            return;
-          }
-          if (pgdump.exitCode !== 0) {
-            this.logger.error(`Gzip exited with code 0 but pgdump exited with ${pgdump.exitCode}`);
-            return;
-          }
-          resolve();
-        });
-      });
-      await this.storageRepository.rename(backupFilePath, backupFilePath.replace('.tmp', ''));
+      await createDatabaseBackup(this.backupRepos);
    } catch (error) {
-      this.logger.error(`Database Backup Failure: ${error}`);
-      await this.storageRepository
-        .unlink(backupFilePath)
-        .catch((error) => this.logger.error(`Failed to delete failed backup file: ${error}`));
+      if (error instanceof UnsupportedPostgresError) {
+        return JobStatus.Failed;
+      }
+
      throw error;
    }

-    this.logger.log(`Database Backup Success`);
    await this.cleanupDatabaseBackups();
    return JobStatus.Success;
  }
+
+  private get backupRepos() {
+    return {
+      logger: this.logger,
+      storage: this.storageRepository,
+      config: this.configRepository,
+      process: this.processRepository,
+      database: this.databaseRepository,
+    };
+  }
 }
@@ -1,5 +1,5 @@
 import { jwtVerify } from 'jose';
-import { SystemMetadataKey } from 'src/enum';
+import { MaintenanceAction, SystemMetadataKey } from 'src/enum';
 import { CliService } from 'src/services/cli.service';
 import { factory } from 'test/small.factory';
 import { newTestService, ServiceMocks } from 'test/utils';
@@ -95,7 +95,14 @@ describe(CliService.name, () => {
    });

    it('should disable maintenance mode', async () => {
-      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: true, secret: 'secret' });
+      mocks.systemMetadata.get.mockResolvedValue({
+        isMaintenanceMode: true,
+        secret: 'secret',
+        action: {
+          action: MaintenanceAction.Start,
+        },
+      });
+
      await expect(sut.disableMaintenanceMode()).resolves.toEqual({
        alreadyDisabled: false,
      });
@@ -109,7 +116,14 @@ describe(CliService.name, () => {

  describe('enableMaintenanceMode', () => {
    it('should not do anything if in maintenance mode', async () => {
-      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: true, secret: 'secret' });
+      mocks.systemMetadata.get.mockResolvedValue({
+        isMaintenanceMode: true,
+        secret: 'secret',
+        action: {
+          action: MaintenanceAction.Start,
+        },
+      });
+
      await expect(sut.enableMaintenanceMode()).resolves.toEqual(
        expect.objectContaining({
          alreadyEnabled: true,
@@ -133,13 +147,22 @@ describe(CliService.name, () => {
      expect(mocks.systemMetadata.set).toHaveBeenCalledWith(SystemMetadataKey.MaintenanceMode, {
        isMaintenanceMode: true,
        secret: expect.stringMatching(/^\w{128}$/),
+        action: {
+          action: 'start',
+        },
      });
    });

    const RE_LOGIN_URL = /https:\/\/my.immich.app\/maintenance\?token=([A-Za-z0-9-_]*\.[A-Za-z0-9-_]*\.[A-Za-z0-9-_]*)/;

    it('should return a valid login URL', async () => {
-      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: true, secret: 'secret' });
+      mocks.systemMetadata.get.mockResolvedValue({
+        isMaintenanceMode: true,
+        secret: 'secret',
+        action: {
+          action: MaintenanceAction.Start,
+        },
+      });

      const result = await sut.enableMaintenanceMode();

@@ -3,7 +3,7 @@ import { isAbsolute } from 'node:path';
 import { SALT_ROUNDS } from 'src/constants';
 import { MaintenanceAuthDto } from 'src/dtos/maintenance.dto';
 import { UserAdminResponseDto, mapUserAdmin } from 'src/dtos/user.dto';
-import { SystemMetadataKey } from 'src/enum';
+import { MaintenanceAction, SystemMetadataKey } from 'src/enum';
 import { BaseService } from 'src/services/base.service';
 import { createMaintenanceLoginUrl, generateMaintenanceSecret } from 'src/utils/maintenance';
 import { getExternalDomain } from 'src/utils/misc';
@@ -86,6 +86,9 @@ export class CliService extends BaseService {
    await this.systemMetadataRepository.set(SystemMetadataKey.MaintenanceMode, {
      isMaintenanceMode: true,
      secret,
+      action: {
+        action: MaintenanceAction.Start,
+      },
    });

    await this.appRepository.sendOneShotAppRestart({
@@ -0,0 +1,83 @@
+import { BadRequestException } from '@nestjs/common';
+import { DateTime } from 'luxon';
+import { StorageCore } from 'src/cores/storage.core';
+import { StorageFolder } from 'src/enum';
+import { DatabaseBackupService } from 'src/services/database-backup.service';
+import { MaintenanceService } from 'src/services/maintenance.service';
+import { newTestService, ServiceMocks } from 'test/utils';
+
+describe(MaintenanceService.name, () => {
+  let sut: DatabaseBackupService;
+  let mocks: ServiceMocks;
+
+  beforeEach(() => {
+    ({ sut, mocks } = newTestService(DatabaseBackupService));
+  });
+
+  it('should work', () => {
+    expect(sut).toBeDefined();
+  });
+
+  describe('listBackups', () => {
+    it('should give us all backups', async () => {
+      mocks.storage.readdir.mockResolvedValue([
+        `immich-db-backup-${DateTime.fromISO('2025-07-25T11:02:16Z').toFormat("yyyyLLdd'T'HHmmss")}-v1.234.5-pg14.5.sql.gz.tmp`,
+        `immich-db-backup-${DateTime.fromISO('2025-07-27T11:01:16Z').toFormat("yyyyLLdd'T'HHmmss")}-v1.234.5-pg14.5.sql.gz`,
+        'immich-db-backup-1753789649000.sql.gz',
+        `immich-db-backup-${DateTime.fromISO('2025-07-29T11:01:16Z').toFormat("yyyyLLdd'T'HHmmss")}-v1.234.5-pg14.5.sql.gz`,
+      ]);
+      mocks.storage.stat.mockResolvedValue({ size: 1024 } as any);
+
+      await expect(sut.listBackups()).resolves.toMatchObject({
+        backups: [
+          { filename: 'immich-db-backup-20250729T110116-v1.234.5-pg14.5.sql.gz', filesize: 1024 },
+          { filename: 'immich-db-backup-20250727T110116-v1.234.5-pg14.5.sql.gz', filesize: 1024 },
+          { filename: 'immich-db-backup-1753789649000.sql.gz', filesize: 1024 },
+        ],
+      });
+    });
+  });
+
+  describe('deleteBackup', () => {
+    it('should reject invalid file names', async () => {
+      await expect(sut.deleteBackup(['filename'])).rejects.toThrowError(
+        new BadRequestException('Invalid backup name!'),
+      );
+    });
+
+    it('should unlink the target file', async () => {
+      await sut.deleteBackup(['filename.sql']);
+      expect(mocks.storage.unlink).toHaveBeenCalledTimes(1);
+      expect(mocks.storage.unlink).toHaveBeenCalledWith(
+        `${StorageCore.getBaseFolder(StorageFolder.Backups)}/filename.sql`,
+      );
+    });
+  });
+
+  describe('uploadBackup', () => {
+    it('should reject invalid file names', async () => {
+      await expect(sut.uploadBackup({ originalname: 'invalid backup' } as never)).rejects.toThrowError(
+        new BadRequestException('Invalid backup name!'),
+      );
+    });
+
+    it('should write file', async () => {
+      await sut.uploadBackup({ originalname: 'path.sql.gz', buffer: 'buffer' } as never);
+      expect(mocks.storage.createOrOverwriteFile).toBeCalledWith('/data/backups/uploaded-path.sql.gz', 'buffer');
+    });
+  });
+
+  describe('downloadBackup', () => {
+    it('should reject invalid file names', () => {
+      expect(() => sut.downloadBackup('invalid backup')).toThrowError(new BadRequestException('Invalid backup name!'));
+    });
+
+    it('should get backup path', () => {
+      expect(sut.downloadBackup('hello.sql.gz')).toEqual(
+        expect.objectContaining({
+          path: '/data/backups/hello.sql.gz',
+        }),
+      );
+    });
+  });
+});
@@ -0,0 +1,43 @@
+import { Injectable } from '@nestjs/common';
+import { DatabaseBackupListResponseDto } from 'src/dtos/database-backup.dto';
+import { BaseService } from 'src/services/base.service';
+import {
+  deleteDatabaseBackup,
+  downloadDatabaseBackup,
+  listDatabaseBackups,
+  uploadDatabaseBackup,
+} from 'src/utils/database-backups';
+import { ImmichFileResponse } from 'src/utils/file';
+
+/**
+ * This service is available outside of maintenance mode to manage maintenance mode
+ */
+@Injectable()
+export class DatabaseBackupService extends BaseService {
+  async listBackups(): Promise<DatabaseBackupListResponseDto> {
+    const backups = await listDatabaseBackups(this.backupRepos);
+    return { backups };
+  }
+
+  deleteBackup(files: string[]): Promise<void> {
+    return deleteDatabaseBackup(this.backupRepos, files);
+  }
+
+  async uploadBackup(file: Express.Multer.File): Promise<void> {
+    return uploadDatabaseBackup(this.backupRepos, file);
+  }
+
+  downloadBackup(fileName: string): ImmichFileResponse {
+    return downloadDatabaseBackup(fileName);
+  }
+
+  private get backupRepos() {
+    return {
+      logger: this.logger,
+      storage: this.storageRepository,
+      config: this.configRepository,
+      process: this.processRepository,
+      database: this.databaseRepository,
+    };
+  }
+}
@@ -9,6 +9,7 @@ import { AuthAdminService } from 'src/services/auth-admin.service';
 import { AuthService } from 'src/services/auth.service';
 import { BackupService } from 'src/services/backup.service';
 import { CliService } from 'src/services/cli.service';
+import { DatabaseBackupService } from 'src/services/database-backup.service';
 import { DatabaseService } from 'src/services/database.service';
 import { DownloadService } from 'src/services/download.service';
 import { DuplicateService } from 'src/services/duplicate.service';
@@ -59,6 +60,7 @@ export const services = [
  AuthAdminService,
  BackupService,
  CliService,
+  DatabaseBackupService,
  DatabaseService,
  DownloadService,
  DuplicateService,
@@ -1,4 +1,4 @@
-import { SystemMetadataKey } from 'src/enum';
+import { MaintenanceAction, SystemMetadataKey } from 'src/enum';
 import { MaintenanceService } from 'src/services/maintenance.service';
 import { newTestService, ServiceMocks } from 'test/utils';

@@ -36,28 +36,96 @@ describe(MaintenanceService.name, () => {
    });

    it('should return true if enabled', async () => {
-      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: true, secret: '' });
+      mocks.systemMetadata.get.mockResolvedValue({
+        isMaintenanceMode: true,
+        secret: '',
+        action: { action: MaintenanceAction.Start },
+      });

      await expect(sut.getMaintenanceMode()).resolves.toEqual({
        isMaintenanceMode: true,
        secret: '',
+        action: {
+          action: 'start',
+        },
      });

      expect(mocks.systemMetadata.get).toHaveBeenCalled();
    });
  });

+  describe('integrityCheck', () => {
+    it('generate integrity report', async () => {
+      mocks.storage.readdir.mockResolvedValue(['.immich', 'file1', 'file2']);
+      mocks.storage.readFile.mockResolvedValue(undefined as never);
+      mocks.storage.overwriteFile.mockRejectedValue(undefined as never);
+
+      await expect(sut.detectPriorInstall()).resolves.toMatchInlineSnapshot(`
+        {
+          "storage": [
+            {
+              "files": 2,
+              "folder": "encoded-video",
+              "readable": true,
+              "writable": false,
+            },
+            {
+              "files": 2,
+              "folder": "library",
+              "readable": true,
+              "writable": false,
+            },
+            {
+              "files": 2,
+              "folder": "upload",
+              "readable": true,
+              "writable": false,
+            },
+            {
+              "files": 2,
+              "folder": "profile",
+              "readable": true,
+              "writable": false,
+            },
+            {
+              "files": 2,
+              "folder": "thumbs",
+              "readable": true,
+              "writable": false,
+            },
+            {
+              "files": 2,
+              "folder": "backups",
+              "readable": true,
+              "writable": false,
+            },
+          ],
+        }
+      `);
+    });
+  });
+
  describe('startMaintenance', () => {
    it('should set maintenance mode and return a secret', async () => {
      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: false });

-      await expect(sut.startMaintenance('admin')).resolves.toMatchObject({
+      await expect(
+        sut.startMaintenance(
+          {
+            action: MaintenanceAction.Start,
+          },
+          'admin',
+        ),
+      ).resolves.toMatchObject({
        jwt: expect.any(String),
      });

      expect(mocks.systemMetadata.set).toHaveBeenCalledWith(SystemMetadataKey.MaintenanceMode, {
        isMaintenanceMode: true,
        secret: expect.stringMatching(/^\w{128}$/),
+        action: {
+          action: 'start',
+        },
      });

      expect(mocks.event.emit).toHaveBeenCalledWith('AppRestart', {
@@ -78,7 +146,13 @@ describe(MaintenanceService.name, () => {
    });

    it('should generate a login url with JWT', async () => {
-      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: true, secret: 'secret' });
+      mocks.systemMetadata.get.mockResolvedValue({
+        isMaintenanceMode: true,
+        secret: 'secret',
+        action: {
+          action: MaintenanceAction.Start,
+        },
+      });

      await expect(
        sut.createLoginUrl({
@@ -1,11 +1,21 @@
-import { Injectable } from '@nestjs/common';
+import { BadRequestException, Injectable } from '@nestjs/common';
 import { OnEvent } from 'src/decorators';
-import { MaintenanceAuthDto } from 'src/dtos/maintenance.dto';
-import { SystemMetadataKey } from 'src/enum';
+import {
+  MaintenanceAuthDto,
+  MaintenanceDetectInstallResponseDto,
+  MaintenanceStatusResponseDto,
+  SetMaintenanceModeDto,
+} from 'src/dtos/maintenance.dto';
+import { MaintenanceAction, SystemMetadataKey } from 'src/enum';
 import { ArgOf } from 'src/repositories/event.repository';
 import { BaseService } from 'src/services/base.service';
 import { MaintenanceModeState } from 'src/types';
-import { createMaintenanceLoginUrl, generateMaintenanceSecret, signMaintenanceJwt } from 'src/utils/maintenance';
+import {
+  createMaintenanceLoginUrl,
+  detectPriorInstall,
+  generateMaintenanceSecret,
+  signMaintenanceJwt,
+} from 'src/utils/maintenance';
 import { getExternalDomain } from 'src/utils/misc';

 /**
@@ -19,9 +29,25 @@ export class MaintenanceService extends BaseService {
      .then((state) => state ?? { isMaintenanceMode: false });
  }

-  async startMaintenance(username: string): Promise<{ jwt: string }> {
+  getMaintenanceStatus(): MaintenanceStatusResponseDto {
+    return {
+      active: false,
+      action: MaintenanceAction.End,
+    };
+  }
+
+  detectPriorInstall(): Promise<MaintenanceDetectInstallResponseDto> {
+    return detectPriorInstall(this.storageRepository);
+  }
+
+  async startMaintenance(action: SetMaintenanceModeDto, username: string): Promise<{ jwt: string }> {
    const secret = generateMaintenanceSecret();
-    await this.systemMetadataRepository.set(SystemMetadataKey.MaintenanceMode, { isMaintenanceMode: true, secret });
+    await this.systemMetadataRepository.set(SystemMetadataKey.MaintenanceMode, {
+      isMaintenanceMode: true,
+      secret,
+      action,
+    });
+
    await this.eventRepository.emit('AppRestart', { isMaintenanceMode: true });

    return {
@@ -31,6 +57,20 @@ export class MaintenanceService extends BaseService {
    };
  }

+  async startRestoreFlow(): Promise<{ jwt: string }> {
+    const adminUser = await this.userRepository.getAdmin();
+    if (adminUser) {
+      throw new BadRequestException('The server already has an admin');
+    }
+
+    return this.startMaintenance(
+      {
+        action: MaintenanceAction.SelectDatabaseRestore,
+      },
+      'admin',
+    );
+  }
+
  @OnEvent({ name: 'AppRestart', server: true })
  onRestart(event: ArgOf<'AppRestart'>, ack?: (ok: 'ok') => void): void {
    this.logger.log(`Restarting due to event... ${JSON.stringify(event)}`);