feat: maintenance mode (#23431)

* feat: add a `maintenance.enabled` config flag * feat: implement graceful restart feat: restart when maintenance config is toggled * feat: boot a stripped down maintenance api if enabled * feat: cli command to toggle maintenance mode * chore: fallback IMMICH_SERVER_URL environment variable in process * chore: add additional routes to maintenance controller * fix: don't wait for nest application to close to finish request response * chore: add a failsafe on restart to prevent other exit codes from preventing restart * feat: redirect into/from maintenance page * refactor: use system metadata for maintenance status * refactor: wait on WebSocket connection to refresh * feat: broadcast websocket event on server restart refactor: listen to WS instead of polling * refactor: bubble up maintenance information instead of hijacking in fetch function feat: show modal when server is restarting * chore: increase timeout for ungraceful restart * refactor: deduplicate code between api/maintenance workers * fix: skip config check if database is not initialised * fix: add `maintenanceMode` field to system config test * refactor: move maintenance resolution code to static method in service * chore: clean up linter issues * chore: generate dart openapi * refactor: use try{} block for maintenance mode check * fix: logic error in server redirect * chore: include `maintenanceMode` key in e2e test * chore: add i18n entries for maintenance screens * chore: remove negated condition from hook * fix: should set default value not override in service * fix: minor error in page * feat: initial draft of maintenance module, repo., worker controller, worker service * refactor: move broadcast code into notification service * chore: connect websocket on client if in maintenance * chore: set maintenance module app name * refactor: rename repository to include worker chore: configure websocket adapter * feat: reimplement maintenance mode exit with new module * refactor: add a constant enum for ExitCode * refactor: remove redundant route for maintenance * refactor: only spin up kysely on boot (rather than a Nest app) * refactor(web): move redirect logic into +layout file where modal is setup * feat: add Maintenance permission * refactor: merge common code between api/maintenance * fix: propagate changes from the CLI to servers * feat: maintenance authentication guard * refactor: unify maintenance code into repository feat: add a step to generate maintenance mode token * feat: jwt auth for maintenance * refactor: switch from nest jwt to just jsonwebtokens * feat: log into maintenance mode from CLI command * refactor: use `secret` instead of `token` in jwt terminology chore: log maintenance mode login URL on boot chore: don't make CLI actions reload if already in target state * docs: initial draft for maintenance mode page * refactor: always validate the maintenance auth on the server * feat: add a link to maintenance mode documentation * feat: redirect users back to the last page they were on when exiting maintenance * refactor: provide closeFn in both maintenance repos. * refactor: ensure the user is also redirected by the server * chore: swap jsonwebtoken for jose * refactor: introduce AppRestartEvent w/o secret passing * refactor: use navigation goto * refactor: use `continue` instead of `next` * chore: lint fixes for server * chore: lint fixes for web * test: add mock for maintenance repository * test: add base service dependency to maintenance * chore: remove @types/jsonwebtoken * refactor: close database connection after startup check * refactor: use `request#auth` key * refactor: use service instead of repository chore: read token from cookie if possible chore: rename client event to AppRestartV1 * refactor: more concise redirect logic on web * refactor: move redirect check into utils refactor: update translation strings to be more sensible * refactor: always validate login (i.e. check cookie) * refactor: lint, open-api, remove old dto * refactor: encode at point of usage * refactor: remove business logic from repositories * chore: fix server/web lints * refactor: remove repository mock * chore: fix formatting * test: write service mocks for maintenance mode * test: write cli service tests * fix: catch errors when closing app * fix: always report no maintenance when usual API is available * test: api e2e maintenance spec * chore: add response builder * chore: add helper to set maint. auth cookie * feat: add SSR to maintenance API * test(e2e): write web spec for maintenance * chore: clean up lint issues * chore: format files * feat: perform 302 redirect at server level during maintenance * fix: keep trying to stop immich until it succeeds (CLI issue) * chore: lint/format * refactor: annotate references to other services in worker service * chore: lint * refactor: remove unnecessary await Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com> * refactor: move static methods into util * refactor: assert secret exists in maintenance worker * refactor: remove assertion which isn't necessary anymore * refactor: remove assertion * refactor: remove outer try {} catch block from loadMaintenanceAuth * refactor: undo earlier change to vite.config.ts * chore: update tests due to refactors * revert: vite.config.ts * test: expect string jwt * chore: move blanket exceptions into controllers * test: update tests according with last change * refactor: use respondWithCookie refactor: merge start/end into one route refactor: rename MaintenanceRepository to AppRepository chore: use new ApiTag/Endpoint refactor: apply other requested changes * chore: regenerate openapi * chore: lint/format * chore: remove secureOnly for maint. cookie * refactor: move maintenance worker code into src/maintenance\nfix: various test fixes * refactor: use `action` property for setting maint. mode * refactor: remove Websocket#restartApp in favour of individual methods * chore: incomplete commit * chore: remove stray log * fix: call exitApp from maintenance worker on exit * fix: add app repository mock * fix: ensure maintenance cookies are secure * fix: run playwright tests over secure context (localhost) * test: update other references to 127.0.0.1 * refactor: use serverSideEmitWithAck * chore: correct the logic in tryTerminate * test: juggle cookies ourselves * chore: fix lint error for e2e spec * chore: format e2e test * fix: set cookie secure/non-secure depending on context * chore: format files --------- Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
2026-05-18 03:10:24 +03:00 · 2025-11-17 17:15:44 +00:00
parent ce82e27f4b
commit 15e00f82f0
73 changed files with 2592 additions and 136 deletions
@@ -0,0 +1,58 @@
+import {
+  CanActivate,
+  ExecutionContext,
+  Injectable,
+  SetMetadata,
+  applyDecorators,
+  createParamDecorator,
+} from '@nestjs/common';
+import { Reflector } from '@nestjs/core';
+import { Request } from 'express';
+import { MaintenanceAuthDto } from 'src/dtos/maintenance.dto';
+import { MetadataKey } from 'src/enum';
+import { MaintenanceWorkerService } from 'src/maintenance/maintenance-worker.service';
+import { LoggingRepository } from 'src/repositories/logging.repository';
+
+export const MaintenanceRoute = (options = {}): MethodDecorator => {
+  const decorators: MethodDecorator[] = [SetMetadata(MetadataKey.AuthRoute, options)];
+  return applyDecorators(...decorators);
+};
+
+export interface MaintenanceAuthRequest extends Request {
+  auth?: MaintenanceAuthDto;
+}
+
+export interface MaintenanceAuthenticatedRequest extends Request {
+  auth: MaintenanceAuthDto;
+}
+
+export const MaintenanceAuth = createParamDecorator((data, context: ExecutionContext): MaintenanceAuthDto => {
+  return context.switchToHttp().getRequest<MaintenanceAuthenticatedRequest>().auth;
+});
+
+@Injectable()
+export class MaintenanceAuthGuard implements CanActivate {
+  constructor(
+    private logger: LoggingRepository,
+    private reflector: Reflector,
+    private service: MaintenanceWorkerService,
+  ) {
+    this.logger.setContext(MaintenanceAuthGuard.name);
+  }
+
+  async canActivate(context: ExecutionContext): Promise<boolean> {
+    const targets = [context.getHandler()];
+    const options = this.reflector.getAllAndOverride<{ _emptyObject: never } | undefined>(
+      MetadataKey.AuthRoute,
+      targets,
+    );
+    if (!options) {
+      return true;
+    }
+
+    const request = context.switchToHttp().getRequest<MaintenanceAuthRequest>();
+    request.auth = await this.service.authenticate(request.headers);
+
+    return true;
+  }
+}
@@ -0,0 +1,59 @@
+import { Injectable } from '@nestjs/common';
+import {
+  OnGatewayConnection,
+  OnGatewayDisconnect,
+  OnGatewayInit,
+  WebSocketGateway,
+  WebSocketServer,
+} from '@nestjs/websockets';
+import { Server, Socket } from 'socket.io';
+import { AppRepository } from 'src/repositories/app.repository';
+import { AppRestartEvent, ArgsOf } from 'src/repositories/event.repository';
+import { LoggingRepository } from 'src/repositories/logging.repository';
+
+export const serverEvents = ['AppRestart'] as const;
+export type ServerEvents = (typeof serverEvents)[number];
+
+export interface ClientEventMap {
+  AppRestartV1: [AppRestartEvent];
+}
+
+@WebSocketGateway({
+  cors: true,
+  path: '/api/socket.io',
+  transports: ['websocket'],
+})
+@Injectable()
+export class MaintenanceWebsocketRepository implements OnGatewayConnection, OnGatewayDisconnect, OnGatewayInit {
+  @WebSocketServer()
+  private websocketServer?: Server;
+
+  constructor(
+    private logger: LoggingRepository,
+    private appRepository: AppRepository,
+  ) {
+    this.logger.setContext(MaintenanceWebsocketRepository.name);
+  }
+
+  afterInit(websocketServer: Server) {
+    this.logger.log('Initialized websocket server');
+    websocketServer.on('AppRestart', () => this.appRepository.exitApp());
+  }
+
+  clientBroadcast<T extends keyof ClientEventMap>(event: T, ...data: ClientEventMap[T]) {
+    this.websocketServer?.emit(event, ...data);
+  }
+
+  serverSend<T extends ServerEvents>(event: T, ...args: ArgsOf<T>): void {
+    this.logger.debug(`Server event: ${event} (send)`);
+    this.websocketServer?.serverSideEmit(event, ...args);
+  }
+
+  handleConnection(client: Socket) {
+    this.logger.log(`Websocket Connect:    ${client.id}`);
+  }
+
+  handleDisconnect(client: Socket) {
+    this.logger.log(`Websocket Disconnect: ${client.id}`);
+  }
+}
@@ -0,0 +1,43 @@
+import { Body, Controller, Get, Post, Req, Res } from '@nestjs/common';
+import { Request, Response } from 'express';
+import { MaintenanceAuthDto, MaintenanceLoginDto, SetMaintenanceModeDto } from 'src/dtos/maintenance.dto';
+import { ServerConfigDto } from 'src/dtos/server.dto';
+import { ImmichCookie, MaintenanceAction } from 'src/enum';
+import { MaintenanceRoute } from 'src/maintenance/maintenance-auth.guard';
+import { MaintenanceWorkerService } from 'src/maintenance/maintenance-worker.service';
+import { GetLoginDetails } from 'src/middleware/auth.guard';
+import { LoginDetails } from 'src/services/auth.service';
+import { respondWithCookie } from 'src/utils/response';
+
+@Controller()
+export class MaintenanceWorkerController {
+  constructor(private service: MaintenanceWorkerService) {}
+
+  @Get('server/config')
+  getServerConfig(): Promise<ServerConfigDto> {
+    return this.service.getSystemConfig();
+  }
+
+  @Post('admin/maintenance/login')
+  async maintenanceLogin(
+    @Req() request: Request,
+    @Body() dto: MaintenanceLoginDto,
+    @GetLoginDetails() loginDetails: LoginDetails,
+    @Res({ passthrough: true }) res: Response,
+  ): Promise<MaintenanceAuthDto> {
+    const token = dto.token ?? request.cookies[ImmichCookie.MaintenanceToken];
+    const auth = await this.service.login(token);
+    return respondWithCookie(res, auth, {
+      isSecure: loginDetails.isSecure,
+      values: [{ key: ImmichCookie.MaintenanceToken, value: token }],
+    });
+  }
+
+  @Post('admin/maintenance')
+  @MaintenanceRoute()
+  async setMaintenanceMode(@Body() dto: SetMaintenanceModeDto): Promise<void> {
+    if (dto.action === MaintenanceAction.End) {
+      await this.service.endMaintenance();
+    }
+  }
+}
@@ -0,0 +1,128 @@
+import { UnauthorizedException } from '@nestjs/common';
+import { SignJWT } from 'jose';
+import { SystemMetadataKey } from 'src/enum';
+import { MaintenanceWebsocketRepository } from 'src/maintenance/maintenance-websocket.repository';
+import { MaintenanceWorkerService } from 'src/maintenance/maintenance-worker.service';
+import { automock, getMocks, ServiceMocks } from 'test/utils';
+
+describe(MaintenanceWorkerService.name, () => {
+  let sut: MaintenanceWorkerService;
+  let mocks: ServiceMocks;
+  let maintenanceWorkerRepositoryMock: MaintenanceWebsocketRepository;
+
+  beforeEach(() => {
+    mocks = getMocks();
+    maintenanceWorkerRepositoryMock = automock(MaintenanceWebsocketRepository, { args: [mocks.logger], strict: false });
+    sut = new MaintenanceWorkerService(
+      mocks.logger as never,
+      mocks.app,
+      mocks.config,
+      mocks.systemMetadata as never,
+      maintenanceWorkerRepositoryMock,
+    );
+  });
+
+  it('should work', () => {
+    expect(sut).toBeDefined();
+  });
+
+  describe('getSystemConfig', () => {
+    it('should respond the server is in maintenance mode', async () => {
+      await expect(sut.getSystemConfig()).resolves.toMatchObject(
+        expect.objectContaining({
+          maintenanceMode: true,
+        }),
+      );
+
+      expect(mocks.systemMetadata.get).toHaveBeenCalled();
+    });
+  });
+
+  describe('logSecret', () => {
+    const RE_LOGIN_URL = /https:\/\/my.immich.app\/maintenance\?token=([A-Za-z0-9-_]*\.[A-Za-z0-9-_]*\.[A-Za-z0-9-_]*)/;
+
+    it('should log a valid login URL', async () => {
+      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: true, secret: 'secret' });
+      await expect(sut.logSecret()).resolves.toBeUndefined();
+      expect(mocks.logger.log).toHaveBeenCalledWith(expect.stringMatching(RE_LOGIN_URL));
+
+      const [url] = mocks.logger.log.mock.lastCall!;
+      const token = RE_LOGIN_URL.exec(url)![1];
+
+      await expect(sut.login(token)).resolves.toEqual(
+        expect.objectContaining({
+          username: 'immich-admin',
+        }),
+      );
+    });
+  });
+
+  describe('authenticate', () => {
+    it('should fail without a cookie', async () => {
+      await expect(sut.authenticate({})).rejects.toThrowError(new UnauthorizedException('Missing JWT Token'));
+    });
+
+    it('should parse cookie properly', async () => {
+      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: true, secret: 'secret' });
+
+      await expect(
+        sut.authenticate({
+          cookie: 'immich_maintenance_token=invalid-jwt',
+        }),
+      ).rejects.toThrowError(new UnauthorizedException('Invalid JWT Token'));
+    });
+  });
+
+  describe('login', () => {
+    it('should fail without token', async () => {
+      await expect(sut.login()).rejects.toThrowError(new UnauthorizedException('Missing JWT Token'));
+    });
+
+    it('should fail with expired JWT', async () => {
+      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: true, secret: 'secret' });
+
+      const jwt = await new SignJWT({})
+        .setProtectedHeader({ alg: 'HS256' })
+        .setIssuedAt()
+        .setExpirationTime('0s')
+        .sign(new TextEncoder().encode('secret'));
+
+      await expect(sut.login(jwt)).rejects.toThrowError(new UnauthorizedException('Invalid JWT Token'));
+    });
+
+    it('should succeed with valid JWT', async () => {
+      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: true, secret: 'secret' });
+
+      const jwt = await new SignJWT({ _mockValue: true })
+        .setProtectedHeader({ alg: 'HS256' })
+        .setIssuedAt()
+        .setExpirationTime('4h')
+        .sign(new TextEncoder().encode('secret'));
+
+      await expect(sut.login(jwt)).resolves.toEqual(
+        expect.objectContaining({
+          _mockValue: true,
+        }),
+      );
+    });
+  });
+
+  describe('endMaintenance', () => {
+    it('should set maintenance mode', async () => {
+      mocks.systemMetadata.get.mockResolvedValue({ isMaintenanceMode: false });
+      await expect(sut.endMaintenance()).resolves.toBeUndefined();
+
+      expect(mocks.systemMetadata.set).toHaveBeenCalledWith(SystemMetadataKey.MaintenanceMode, {
+        isMaintenanceMode: false,
+      });
+
+      expect(maintenanceWorkerRepositoryMock.clientBroadcast).toHaveBeenCalledWith('AppRestartV1', {
+        isMaintenanceMode: false,
+      });
+
+      expect(maintenanceWorkerRepositoryMock.serverSend).toHaveBeenCalledWith('AppRestart', {
+        isMaintenanceMode: false,
+      });
+    });
+  });
+});
@@ -0,0 +1,161 @@
+import { Injectable, UnauthorizedException } from '@nestjs/common';
+import { parse } from 'cookie';
+import { NextFunction, Request, Response } from 'express';
+import { jwtVerify } from 'jose';
+import { readFileSync } from 'node:fs';
+import { IncomingHttpHeaders } from 'node:http';
+import { MaintenanceAuthDto } from 'src/dtos/maintenance.dto';
+import { ImmichCookie, SystemMetadataKey } from 'src/enum';
+import { MaintenanceWebsocketRepository } from 'src/maintenance/maintenance-websocket.repository';
+import { AppRepository } from 'src/repositories/app.repository';
+import { ConfigRepository } from 'src/repositories/config.repository';
+import { LoggingRepository } from 'src/repositories/logging.repository';
+import { SystemMetadataRepository } from 'src/repositories/system-metadata.repository';
+import { type ApiService as _ApiService } from 'src/services/api.service';
+import { type BaseService as _BaseService } from 'src/services/base.service';
+import { type ServerService as _ServerService } from 'src/services/server.service';
+import { MaintenanceModeState } from 'src/types';
+import { getConfig } from 'src/utils/config';
+import { createMaintenanceLoginUrl } from 'src/utils/maintenance';
+import { getExternalDomain } from 'src/utils/misc';
+
+/**
+ * This service is available inside of maintenance mode to manage maintenance mode
+ */
+@Injectable()
+export class MaintenanceWorkerService {
+  constructor(
+    protected logger: LoggingRepository,
+    private appRepository: AppRepository,
+    private configRepository: ConfigRepository,
+    private systemMetadataRepository: SystemMetadataRepository,
+    private maintenanceWorkerRepository: MaintenanceWebsocketRepository,
+  ) {
+    this.logger.setContext(this.constructor.name);
+  }
+
+  /**
+   * {@link _BaseService.configRepos}
+   */
+  private get configRepos() {
+    return {
+      configRepo: this.configRepository,
+      metadataRepo: this.systemMetadataRepository,
+      logger: this.logger,
+    };
+  }
+
+  /**
+   * {@link _BaseService.prototype.getConfig}
+   */
+  private getConfig(options: { withCache: boolean }) {
+    return getConfig(this.configRepos, options);
+  }
+
+  /**
+   * {@link _ServerService.getSystemConfig}
+   */
+  async getSystemConfig() {
+    const config = await this.getConfig({ withCache: false });
+
+    return {
+      loginPageMessage: config.server.loginPageMessage,
+      trashDays: config.trash.days,
+      userDeleteDelay: config.user.deleteDelay,
+      oauthButtonText: config.oauth.buttonText,
+      isInitialized: true,
+      isOnboarded: true,
+      externalDomain: config.server.externalDomain,
+      publicUsers: config.server.publicUsers,
+      mapDarkStyleUrl: config.map.darkStyle,
+      mapLightStyleUrl: config.map.lightStyle,
+      maintenanceMode: true,
+    };
+  }
+
+  /**
+   * {@link _ApiService.ssr}
+   */
+  ssr(excludePaths: string[]) {
+    const { resourcePaths } = this.configRepository.getEnv();
+
+    let index = '';
+    try {
+      index = readFileSync(resourcePaths.web.indexHtml).toString();
+    } catch {
+      this.logger.warn(`Unable to open ${resourcePaths.web.indexHtml}, skipping SSR.`);
+    }
+
+    return (request: Request, res: Response, next: NextFunction) => {
+      if (
+        request.url.startsWith('/api') ||
+        request.method.toLowerCase() !== 'get' ||
+        excludePaths.some((item) => request.url.startsWith(item))
+      ) {
+        return next();
+      }
+
+      const maintenancePath = '/maintenance';
+      if (!request.url.startsWith(maintenancePath)) {
+        const params = new URLSearchParams();
+        params.set('continue', request.path);
+        return res.redirect(`${maintenancePath}?${params}`);
+      }
+
+      res.status(200).type('text/html').header('Cache-Control', 'no-store').send(index);
+    };
+  }
+
+  private async secret(): Promise<string> {
+    const state = (await this.systemMetadataRepository.get(SystemMetadataKey.MaintenanceMode)) as {
+      secret: string;
+    };
+
+    return state.secret;
+  }
+
+  async logSecret(): Promise<void> {
+    const { server } = await this.getConfig({ withCache: true });
+
+    const baseUrl = getExternalDomain(server);
+    const url = await createMaintenanceLoginUrl(
+      baseUrl,
+      {
+        username: 'immich-admin',
+      },
+      await this.secret(),
+    );
+
+    this.logger.log(`\n\n🚧 Immich is in maintenance mode, you can log in using the following URL:\n${url}\n`);
+  }
+
+  async authenticate(headers: IncomingHttpHeaders): Promise<MaintenanceAuthDto> {
+    const jwtToken = parse(headers.cookie || '')[ImmichCookie.MaintenanceToken];
+    return this.login(jwtToken);
+  }
+
+  async login(jwt?: string): Promise<MaintenanceAuthDto> {
+    if (!jwt) {
+      throw new UnauthorizedException('Missing JWT Token');
+    }
+
+    const secret = await this.secret();
+
+    try {
+      const result = await jwtVerify<MaintenanceAuthDto>(jwt, new TextEncoder().encode(secret));
+      return result.payload;
+    } catch {
+      throw new UnauthorizedException('Invalid JWT Token');
+    }
+  }
+
+  async endMaintenance(): Promise<void> {
+    const state: MaintenanceModeState = { isMaintenanceMode: false as const };
+    await this.systemMetadataRepository.set(SystemMetadataKey.MaintenanceMode, state);
+
+    // => corresponds to notification.service.ts#onAppRestart
+    this.maintenanceWorkerRepository.clientBroadcast('AppRestartV1', state);
+    this.maintenanceWorkerRepository.serverSend('AppRestart', state);
+    this.appRepository.exitApp();
+  }
+}