mirror of
https://github.com/immich-app/immich.git
synced 2026-05-18 03:10:24 +03:00
feat(server): resolve duplicates (#25316)
* feat(web): Synchronize information from deduplicated images * Added new settings menu to the the deduplication tab. * The toggable options in the settings are synchronization of: albums, favorites, ratings, description, visibility and location. * When synchronizing the albums, the resolved images will be added to all albums of the duplicates. * When synchronizing the favorite status, the resolved images will be marked as favorite, if at least one selectable image is marked as favorite. * When synchronizing the ratings, the highest rating from the selectable images will be applied to the resolved image. * When synchronizing the description, all descriptions from the selectable images will be merged into one description for the resolved image. * When synchronizing the visibility, the most restrictive visibility setting from the selectable images will be applied to the resolved image. * When synchronizing the location, if exactly one unique location exists among the selectable images, this location will be applied to the resolved image. * There is no additional UI for these settings to keep the visual clutter minimal. The settings are applied automatically based on the user's preferences. * Replace addAssetToAlbums with copyAsset * fix linter * feat(web): add duplicate sync fields and fix typo * feat(web): add tag sync and enhance duplicate resolution This update introduces tag synchronization for duplicate resolution, ensuring all unique tag IDs from duplicates are applied to kept assets. The visibility sync logic is updated to use a simplified ordering, as the hidden status items will never show up in a duplicate set. Album synchronization now merges albums directly via addAssetsToAlbums; as the approach with copyAsset API endpoint was ineffiecient. Description, rating, and location sync logic is improved for correctness. and deduplication. i18n strings were added / updated. * feat(server): move duplicate resolution to backend with sync and stacking Moves duplicate metadata synchronization from frontend to backend, enabling robust batch operations and proper validation. This is an improved refactor of PR #13851. New endpoints: - POST /duplicates/resolve - batch resolve with configurable metadata sync - POST /duplicates/stack - create stacks from duplicate groups - GET /duplicates - now includes suggestedKeepAssetIds based on file size and EXIF Key changes: - Move sync logic (albums, tags, favorites, ratings, descriptions, location, visibility) to server - Add server-side metadata merge policies with proper conflict resolution - Replace client-side resolution logic with new backend endpoints - Add comprehensive E2E tests (70+ test cases) and unit tests - Update OpenAPI specs and TypeScript SDK No breaking changes - only additions to existing API. * feat(preferences): enable all duplicate sync settings by default * chore: clean up * chore: clean up * refactor: rename & clean up * fix: preference upgrade * chore: linting * refactor(e2e): use updateAssets API for setAssetDuplicateId * fix: visibility sync logic in duplicate resolution * fix(duplicate): write description to exifUpdate Previously the duplicate resolution populated assetUpdate.description even though description belongs to exif info. * fix(duplicate): remove redundant updateLockedColumns wrapper updateAllExif already computes lockedProperties via distinctLocked using Object.keys(options). The wrapper added a lockedProperties key to the options object, causing the spurious string 'lockedProperties' to be stored in the lockedProperties array. * fix(duplicate): write merged tags to asset_exif to survive metadata re-extraction During duplicate resolution, replaceAssetTags correctly wrote merged tag IDs to the tag_asset table, but never updated asset_exif.tags or locked the tags property. The subsequent SidecarWrite → AssetExtractMetadata chain calls applyTagList, which destructively replaces tag_asset rows with whatever is in asset_exif.tags — still the original per-asset tags, not the merged set. Write merged tag values to asset_exif.tags via updateAllExif (which also locks the property via distinctLocked), and queue SidecarWrite when tags change so they persist to the sidecar file. * docs(duplicates): clarify location and tag sync behavior * refactor(duplicate): remove sync settings, always sync all metadata on resolve Remove DuplicateSyncSettingsDto and the per-field sync toggles (albums, favorites, rating, description, visibility, location, tags). Duplicate resolution now unconditionally syncs all metadata from trashed assets to kept assets. - Remove DuplicateSyncSettingsDto and settings field from DuplicateResolveDto - Update DuplicateService to always run all sync logic without conditionals - Delete DuplicateSettingsModal.svelte and settings gear button from UI - Remove DuplicateSettings type and duplicateSettings persisted store - Update unit and e2e tests to remove settings from resolve requests * docs: update duplicates utility to reflect automatic metadata sync * docs(web): replace duplicates info modal with link to documentation * chore: clean up * fix: add missing type cast to jsonAgg in duplicate repository getAll * fix: skip persisting rating=0 in duplicate merge to avoid unnecessary sidecar write --------- Co-authored-by: Toni <51962051+EinToni@users.noreply.github.com> Co-authored-by: Jason Rasmussen <jason@rasm.me> Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
This commit is contained in:
Phlogi
@@ -0,0 +1,47 @@
|
||||
import { DuplicateController } from 'src/controllers/duplicate.controller';
|
||||
import { DuplicateService } from 'src/services/duplicate.service';
|
||||
import request from 'supertest';
|
||||
import { factory } from 'test/small.factory';
|
||||
import { ControllerContext, controllerSetup, mockBaseService } from 'test/utils';
|
||||
|
||||
describe(DuplicateController.name, () => {
|
||||
let ctx: ControllerContext;
|
||||
const service = mockBaseService(DuplicateService);
|
||||
|
||||
beforeAll(async () => {
|
||||
ctx = await controllerSetup(DuplicateController, [{ provide: DuplicateService, useValue: service }]);
|
||||
return () => ctx.close();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
service.resetAllMocks();
|
||||
ctx.reset();
|
||||
});
|
||||
|
||||
describe('GET /duplicates', () => {
|
||||
it('should be an authenticated route', async () => {
|
||||
await request(ctx.getHttpServer()).get('/duplicates');
|
||||
expect(ctx.authenticate).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('DELETE /duplicates', () => {
|
||||
it('should be an authenticated route', async () => {
|
||||
await request(ctx.getHttpServer()).delete('/duplicates');
|
||||
expect(ctx.authenticate).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('DELETE /duplicates/:id', () => {
|
||||
it('should be an authenticated route', async () => {
|
||||
await request(ctx.getHttpServer()).delete(`/duplicates/${factory.uuid()}`);
|
||||
expect(ctx.authenticate).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should require a valid uuid', async () => {
|
||||
const { status, body } = await request(ctx.getHttpServer()).delete(`/duplicates/123`);
|
||||
expect(status).toBe(400);
|
||||
expect(body).toEqual(factory.responses.badRequest(['id must be a UUID']));
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,9 +1,9 @@
|
||||
import { Body, Controller, Delete, Get, HttpCode, HttpStatus, Param } from '@nestjs/common';
|
||||
import { Body, Controller, Delete, Get, HttpCode, HttpStatus, Param, Post } from '@nestjs/common';
|
||||
import { ApiTags } from '@nestjs/swagger';
|
||||
import { Endpoint, HistoryBuilder } from 'src/decorators';
|
||||
import { BulkIdsDto } from 'src/dtos/asset-ids.response.dto';
|
||||
import { BulkIdResponseDto, BulkIdsDto } from 'src/dtos/asset-ids.response.dto';
|
||||
import { AuthDto } from 'src/dtos/auth.dto';
|
||||
import { DuplicateResponseDto } from 'src/dtos/duplicate.dto';
|
||||
import { DuplicateResolveDto, DuplicateResponseDto } from 'src/dtos/duplicate.dto';
|
||||
import { ApiTag, Permission } from 'src/enum';
|
||||
import { Auth, Authenticated } from 'src/middleware/auth.guard';
|
||||
import { DuplicateService } from 'src/services/duplicate.service';
|
||||
@@ -48,4 +48,16 @@ export class DuplicateController {
|
||||
deleteDuplicate(@Auth() auth: AuthDto, @Param() { id }: UUIDParamDto): Promise<void> {
|
||||
return this.service.delete(auth, id);
|
||||
}
|
||||
|
||||
@Post('resolve')
|
||||
@HttpCode(HttpStatus.OK)
|
||||
@Authenticated({ permission: Permission.DuplicateDelete })
|
||||
@Endpoint({
|
||||
summary: 'Resolve duplicate groups',
|
||||
description: 'Resolve duplicate groups by synchronizing metadata across assets and deleting/trashing duplicates.',
|
||||
history: new HistoryBuilder().added('v3.0.0').alpha('v3.0.0'),
|
||||
})
|
||||
resolveDuplicates(@Auth() auth: AuthDto, @Body() dto: DuplicateResolveDto): Promise<BulkIdResponseDto[]> {
|
||||
return this.service.resolve(auth, dto);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -23,6 +23,7 @@ export enum BulkIdErrorReason {
|
||||
NO_PERMISSION = 'no_permission',
|
||||
NOT_FOUND = 'not_found',
|
||||
UNKNOWN = 'unknown',
|
||||
VALIDATION = 'validation',
|
||||
}
|
||||
|
||||
export class BulkIdsDto {
|
||||
@@ -37,4 +38,5 @@ export class BulkIdResponseDto {
|
||||
success!: boolean;
|
||||
@ApiPropertyOptional({ description: 'Error reason if failed', enum: BulkIdErrorReason })
|
||||
error?: BulkIdErrorReason;
|
||||
errorMessage?: string;
|
||||
}
|
||||
|
||||
@@ -1,9 +1,35 @@
|
||||
import { ApiProperty } from '@nestjs/swagger';
|
||||
import { Type } from 'class-transformer';
|
||||
import { ArrayMinSize, IsArray, ValidateNested } from 'class-validator';
|
||||
import { AssetResponseDto } from 'src/dtos/asset-response.dto';
|
||||
import { ValidateUUID } from 'src/validation';
|
||||
|
||||
export class DuplicateResponseDto {
|
||||
@ApiProperty({ description: 'Duplicate group ID' })
|
||||
duplicateId!: string;
|
||||
@ApiProperty({ description: 'Duplicate assets' })
|
||||
assets!: AssetResponseDto[];
|
||||
|
||||
@ValidateUUID({ each: true, description: 'Suggested asset IDs to keep based on file size and EXIF data' })
|
||||
suggestedKeepAssetIds!: string[];
|
||||
}
|
||||
|
||||
export class DuplicateResolveGroupDto {
|
||||
@ValidateUUID()
|
||||
duplicateId!: string;
|
||||
|
||||
@ValidateUUID({ each: true, description: 'Asset IDs to keep' })
|
||||
keepAssetIds!: string[];
|
||||
|
||||
@ValidateUUID({ each: true, description: 'Asset IDs to trash or delete' })
|
||||
trashAssetIds!: string[];
|
||||
}
|
||||
|
||||
export class DuplicateResolveDto {
|
||||
@ApiProperty({ description: 'List of duplicate groups to resolve' })
|
||||
@ValidateNested({ each: true })
|
||||
@IsArray()
|
||||
@Type(() => DuplicateResolveGroupDto)
|
||||
@ArrayMinSize(1)
|
||||
groups!: DuplicateResolveGroupDto[];
|
||||
}
|
||||
|
||||
@@ -160,6 +160,16 @@ where
|
||||
"session"."userId" = $1
|
||||
and "session"."id" in ($2)
|
||||
|
||||
-- AccessRepository.duplicate.checkOwnerAccess
|
||||
select
|
||||
"asset"."duplicateId"
|
||||
from
|
||||
"asset"
|
||||
where
|
||||
"asset"."duplicateId" in ($1)
|
||||
and "asset"."ownerId" = $2
|
||||
and "asset"."deletedAt" is null
|
||||
|
||||
-- AccessRepository.memory.checkOwnerAccess
|
||||
select
|
||||
"memory"."id"
|
||||
|
||||
@@ -164,6 +164,28 @@ order by
|
||||
"album"."createdAt" desc,
|
||||
"album"."createdAt" desc
|
||||
|
||||
-- AlbumRepository.getByAssetIds
|
||||
select
|
||||
"album"."id",
|
||||
"album_asset"."assetId"
|
||||
from
|
||||
"album"
|
||||
inner join "album_asset" on "album_asset"."albumId" = "album"."id"
|
||||
where
|
||||
(
|
||||
"album"."ownerId" = $1
|
||||
or exists (
|
||||
select
|
||||
from
|
||||
"album_user"
|
||||
where
|
||||
"album_user"."albumId" = "album"."id"
|
||||
and "album_user"."userId" = $2
|
||||
)
|
||||
)
|
||||
and "album_asset"."assetId" in ($3)
|
||||
and "album"."deletedAt" is null
|
||||
|
||||
-- AlbumRepository.getMetadataForIds
|
||||
select
|
||||
"album_asset"."albumId" as "albumId",
|
||||
|
||||
@@ -15,7 +15,26 @@ with
|
||||
inner join lateral (
|
||||
select
|
||||
"asset".*,
|
||||
"asset_exif" as "exifInfo"
|
||||
to_json("asset_exif") as "exifInfo",
|
||||
(
|
||||
select
|
||||
coalesce(json_agg(agg), '[]')
|
||||
from
|
||||
(
|
||||
select
|
||||
"tag"."id",
|
||||
"tag"."value",
|
||||
"tag"."createdAt",
|
||||
"tag"."updatedAt",
|
||||
"tag"."color",
|
||||
"tag"."parentId"
|
||||
from
|
||||
"tag"
|
||||
inner join "tag_asset" on "tag"."id" = "tag_asset"."tagId"
|
||||
where
|
||||
"tag_asset"."assetId" = "asset"."id"
|
||||
) as agg
|
||||
) as "tags"
|
||||
from
|
||||
"asset_exif"
|
||||
where
|
||||
@@ -29,36 +48,84 @@ with
|
||||
and "asset"."stackId" is null
|
||||
group by
|
||||
"asset"."duplicateId"
|
||||
),
|
||||
"unique" as (
|
||||
select
|
||||
"duplicateId"
|
||||
from
|
||||
"duplicates"
|
||||
where
|
||||
json_array_length("assets") = $2
|
||||
),
|
||||
"removed_unique" as (
|
||||
update "asset"
|
||||
set
|
||||
"duplicateId" = $3
|
||||
from
|
||||
"unique"
|
||||
where
|
||||
"asset"."duplicateId" = "unique"."duplicateId"
|
||||
)
|
||||
select
|
||||
*
|
||||
from
|
||||
"duplicates"
|
||||
where
|
||||
not exists (
|
||||
json_array_length("assets") > $2
|
||||
|
||||
-- DuplicateRepository.cleanupSingletonGroups
|
||||
with
|
||||
"singletons" as (
|
||||
select
|
||||
"duplicateId"
|
||||
from
|
||||
"unique"
|
||||
"asset"
|
||||
where
|
||||
"unique"."duplicateId" = "duplicates"."duplicateId"
|
||||
"ownerId" = $1::uuid
|
||||
and "duplicateId" is not null
|
||||
and "deletedAt" is null
|
||||
and "stackId" is null
|
||||
group by
|
||||
"duplicateId"
|
||||
having
|
||||
count("id") = $2
|
||||
)
|
||||
update "asset"
|
||||
set
|
||||
"duplicateId" = $3
|
||||
from
|
||||
"singletons"
|
||||
where
|
||||
"asset"."duplicateId" = "singletons"."duplicateId"
|
||||
|
||||
-- DuplicateRepository.get
|
||||
select
|
||||
"asset"."duplicateId",
|
||||
json_agg(
|
||||
"asset2"
|
||||
order by
|
||||
"asset"."localDateTime" asc
|
||||
) as "assets"
|
||||
from
|
||||
"asset"
|
||||
inner join lateral (
|
||||
select
|
||||
"asset".*,
|
||||
to_json("asset_exif") as "exifInfo",
|
||||
(
|
||||
select
|
||||
coalesce(json_agg(agg), '[]')
|
||||
from
|
||||
(
|
||||
select
|
||||
"tag"."id",
|
||||
"tag"."value",
|
||||
"tag"."createdAt",
|
||||
"tag"."updatedAt",
|
||||
"tag"."color",
|
||||
"tag"."parentId"
|
||||
from
|
||||
"tag"
|
||||
inner join "tag_asset" on "tag"."id" = "tag_asset"."tagId"
|
||||
where
|
||||
"tag_asset"."assetId" = "asset"."id"
|
||||
) as agg
|
||||
) as "tags"
|
||||
from
|
||||
"asset_exif"
|
||||
where
|
||||
"asset_exif"."assetId" = "asset"."id"
|
||||
) as "asset2" on true
|
||||
where
|
||||
"asset"."visibility" in ('archive', 'timeline')
|
||||
and "asset"."duplicateId" = $1::uuid
|
||||
and "asset"."deletedAt" is null
|
||||
and "asset"."stackId" is null
|
||||
group by
|
||||
"asset"."duplicateId"
|
||||
|
||||
-- DuplicateRepository.delete
|
||||
update "asset"
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
import { Kysely, sql } from 'kysely';
|
||||
import { Kysely, NotNull, sql } from 'kysely';
|
||||
import { InjectKysely } from 'nestjs-kysely';
|
||||
import { ChunkedSet, DummyValue, GenerateSql } from 'src/decorators';
|
||||
import { AlbumUserRole, AssetVisibility } from 'src/enum';
|
||||
@@ -285,6 +285,28 @@ class AuthDeviceAccess {
|
||||
}
|
||||
}
|
||||
|
||||
class DuplicateAccess {
|
||||
constructor(private db: Kysely<DB>) {}
|
||||
|
||||
@GenerateSql({ params: [DummyValue.UUID, DummyValue.UUID_SET] })
|
||||
@ChunkedSet({ paramIndex: 1 })
|
||||
async checkOwnerAccess(userId: string, duplicateIds: Set<string>) {
|
||||
if (duplicateIds.size === 0) {
|
||||
return new Set<string>();
|
||||
}
|
||||
|
||||
return this.db
|
||||
.selectFrom('asset')
|
||||
.select('asset.duplicateId')
|
||||
.where('asset.duplicateId', 'in', [...duplicateIds])
|
||||
.where('asset.ownerId', '=', userId)
|
||||
.where('asset.deletedAt', 'is', null)
|
||||
.$narrowType<{ duplicateId: NotNull }>()
|
||||
.execute()
|
||||
.then((assets) => new Set(assets.map((asset) => asset.duplicateId)));
|
||||
}
|
||||
}
|
||||
|
||||
class NotificationAccess {
|
||||
constructor(private db: Kysely<DB>) {}
|
||||
|
||||
@@ -488,6 +510,7 @@ export class AccessRepository {
|
||||
album: AlbumAccess;
|
||||
asset: AssetAccess;
|
||||
authDevice: AuthDeviceAccess;
|
||||
duplicate: DuplicateAccess;
|
||||
memory: MemoryAccess;
|
||||
notification: NotificationAccess;
|
||||
person: PersonAccess;
|
||||
@@ -503,6 +526,7 @@ export class AccessRepository {
|
||||
this.album = new AlbumAccess(db);
|
||||
this.asset = new AssetAccess(db);
|
||||
this.authDevice = new AuthDeviceAccess(db);
|
||||
this.duplicate = new DuplicateAccess(db);
|
||||
this.memory = new MemoryAccess(db);
|
||||
this.notification = new NotificationAccess(db);
|
||||
this.person = new PersonAccess(db);
|
||||
|
||||
@@ -125,6 +125,44 @@ export class AlbumRepository {
|
||||
.execute();
|
||||
}
|
||||
|
||||
@GenerateSql({ params: [DummyValue.UUID, [DummyValue.UUID]] })
|
||||
@ChunkedSet({ paramIndex: 1 })
|
||||
async getByAssetIds(ownerId: string, assetIds: string[]): Promise<Map<string, string[]>> {
|
||||
if (assetIds.length === 0) {
|
||||
return new Map();
|
||||
}
|
||||
|
||||
const results = await this.db
|
||||
.selectFrom('album')
|
||||
.select('album.id')
|
||||
.innerJoin('album_asset', 'album_asset.albumId', 'album.id')
|
||||
.where((eb) =>
|
||||
eb.or([
|
||||
eb('album.ownerId', '=', ownerId),
|
||||
eb.exists(
|
||||
eb
|
||||
.selectFrom('album_user')
|
||||
.whereRef('album_user.albumId', '=', 'album.id')
|
||||
.where('album_user.userId', '=', ownerId),
|
||||
),
|
||||
]),
|
||||
)
|
||||
.where('album_asset.assetId', 'in', assetIds)
|
||||
.where('album.deletedAt', 'is', null)
|
||||
.select('album_asset.assetId')
|
||||
.execute();
|
||||
|
||||
// Group by assetId
|
||||
const map = new Map<string, string[]>();
|
||||
for (const row of results) {
|
||||
const existing = map.get(row.assetId) ?? [];
|
||||
existing.push(row.id);
|
||||
map.set(row.assetId, existing);
|
||||
}
|
||||
|
||||
return map;
|
||||
}
|
||||
|
||||
@GenerateSql({ params: [[DummyValue.UUID]] })
|
||||
@ChunkedArray()
|
||||
async getMetadataForIds(ids: string[]): Promise<AlbumAssetCount[]> {
|
||||
@@ -339,7 +377,12 @@ export class AlbumRepository {
|
||||
if (values.length === 0) {
|
||||
return;
|
||||
}
|
||||
await this.db.insertInto('album_asset').values(values).execute();
|
||||
await this.db
|
||||
.insertInto('album_asset')
|
||||
.values(values)
|
||||
// Allow idempotent album sync without failing on existing album memberships.
|
||||
.onConflict((oc) => oc.columns(['albumId', 'assetId']).doNothing())
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -1,13 +1,19 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
import { Kysely, NotNull, Selectable, ShallowDehydrateObject, sql } from 'kysely';
|
||||
import { jsonArrayFrom } from 'kysely/helpers/postgres';
|
||||
import { InjectKysely } from 'nestjs-kysely';
|
||||
import { columns } from 'src/database';
|
||||
import { Chunked, DummyValue, GenerateSql } from 'src/decorators';
|
||||
import { MapAsset } from 'src/dtos/asset-response.dto';
|
||||
import { AssetType, VectorIndex } from 'src/enum';
|
||||
import { probes } from 'src/repositories/database.repository';
|
||||
import { DB } from 'src/schema';
|
||||
import { AssetExifTable } from 'src/schema/tables/asset-exif.table';
|
||||
import { anyUuid, asUuid, withDefaultVisibility } from 'src/utils/database';
|
||||
|
||||
// Maximum number of candidate duplicates to return from vector search
|
||||
const DUPLICATE_SEARCH_LIMIT = 64;
|
||||
|
||||
interface DuplicateSearch {
|
||||
assetId: string;
|
||||
embedding: string;
|
||||
@@ -34,20 +40,39 @@ export class DuplicateRepository {
|
||||
qb
|
||||
.selectFrom('asset')
|
||||
.$call(withDefaultVisibility)
|
||||
// Use innerJoinLateral to build a composite object per asset that includes
|
||||
// exifInfo and tags. This "asset2" object is then aggregated via jsonAgg.
|
||||
// Tags must be included here (not via separate joins) so they appear in the
|
||||
// final MapAsset[] output - needed for tag synchronization during resolution.
|
||||
.innerJoinLateral(
|
||||
(qb) =>
|
||||
qb
|
||||
.selectFrom('asset_exif')
|
||||
.selectAll('asset')
|
||||
.select((eb) =>
|
||||
eb.table('asset_exif').$castTo<ShallowDehydrateObject<Selectable<AssetExifTable>>>().as('exifInfo'),
|
||||
eb.fn
|
||||
.toJson('asset_exif')
|
||||
.$castTo<ShallowDehydrateObject<Selectable<AssetExifTable>>>()
|
||||
.as('exifInfo'),
|
||||
)
|
||||
|
||||
.select((eb) =>
|
||||
jsonArrayFrom(
|
||||
eb
|
||||
.selectFrom('tag')
|
||||
.select(columns.tag)
|
||||
.innerJoin('tag_asset', 'tag.id', 'tag_asset.tagId')
|
||||
.whereRef('tag_asset.assetId', '=', 'asset.id'),
|
||||
).as('tags'),
|
||||
)
|
||||
.whereRef('asset_exif.assetId', '=', 'asset.id')
|
||||
.as('asset2'),
|
||||
(join) => join.onTrue(),
|
||||
)
|
||||
.select('asset.duplicateId')
|
||||
.select((eb) => eb.fn.jsonAgg('asset2').orderBy('asset.localDateTime', 'asc').as('assets'))
|
||||
.select((eb) =>
|
||||
eb.fn.jsonAgg('asset2').orderBy('asset.localDateTime', 'asc').$castTo<MapAsset[]>().as('assets'),
|
||||
)
|
||||
.where('asset.ownerId', '=', asUuid(userId))
|
||||
.where('asset.duplicateId', 'is not', null)
|
||||
.$narrowType<{ duplicateId: NotNull }>()
|
||||
@@ -55,29 +80,80 @@ export class DuplicateRepository {
|
||||
.where('asset.stackId', 'is', null)
|
||||
.groupBy('asset.duplicateId'),
|
||||
)
|
||||
.with('unique', (qb) =>
|
||||
qb
|
||||
.selectFrom('duplicates')
|
||||
.select('duplicateId')
|
||||
.where((eb) => eb(eb.fn('json_array_length', ['assets']), '=', 1)),
|
||||
)
|
||||
.with('removed_unique', (qb) =>
|
||||
qb
|
||||
.updateTable('asset')
|
||||
.set({ duplicateId: null })
|
||||
.from('unique')
|
||||
.whereRef('asset.duplicateId', '=', 'unique.duplicateId'),
|
||||
)
|
||||
.selectFrom('duplicates')
|
||||
.selectAll()
|
||||
// TODO: compare with filtering by json_array_length > 1
|
||||
.where(({ not, exists }) =>
|
||||
not(exists((eb) => eb.selectFrom('unique').whereRef('unique.duplicateId', '=', 'duplicates.duplicateId'))),
|
||||
)
|
||||
// Filter out singleton groups (only 1 asset) directly in the query
|
||||
.where((eb) => eb(eb.fn('json_array_length', ['assets']), '>', 1))
|
||||
.execute()
|
||||
);
|
||||
}
|
||||
|
||||
@GenerateSql({ params: [DummyValue.UUID] })
|
||||
async cleanupSingletonGroups(userId: string): Promise<void> {
|
||||
// Remove duplicateId from assets that are the only member of their duplicate group
|
||||
await this.db
|
||||
.with('singletons', (qb) =>
|
||||
qb
|
||||
.selectFrom('asset')
|
||||
.select('duplicateId')
|
||||
.where('ownerId', '=', asUuid(userId))
|
||||
.where('duplicateId', 'is not', null)
|
||||
.$narrowType<{ duplicateId: NotNull }>()
|
||||
.where('deletedAt', 'is', null)
|
||||
.where('stackId', 'is', null)
|
||||
.groupBy('duplicateId')
|
||||
.having((eb) => eb.fn.count('id'), '=', 1),
|
||||
)
|
||||
.updateTable('asset')
|
||||
.set({ duplicateId: null })
|
||||
.from('singletons')
|
||||
.whereRef('asset.duplicateId', '=', 'singletons.duplicateId')
|
||||
.execute();
|
||||
}
|
||||
|
||||
@GenerateSql({ params: [DummyValue.UUID, DummyValue.UUID] })
|
||||
async get(duplicateId: string): Promise<{ duplicateId: string; assets: MapAsset[] } | undefined> {
|
||||
const result = await this.db
|
||||
.selectFrom('asset')
|
||||
.$call(withDefaultVisibility)
|
||||
// Use innerJoinLateral to build a composite object per asset that includes
|
||||
// exifInfo and tags. This "asset2" object is then aggregated via jsonAgg.
|
||||
// Tags must be included here (not via separate joins) so they appear in the
|
||||
// final MapAsset[] output - needed for tag synchronization during resolution.
|
||||
.innerJoinLateral(
|
||||
(qb) =>
|
||||
qb
|
||||
.selectFrom('asset_exif')
|
||||
.selectAll('asset')
|
||||
.select((eb) => eb.fn.toJson('asset_exif').as('exifInfo'))
|
||||
.select((eb) =>
|
||||
jsonArrayFrom(
|
||||
eb
|
||||
.selectFrom('tag')
|
||||
.select(columns.tag)
|
||||
.innerJoin('tag_asset', 'tag.id', 'tag_asset.tagId')
|
||||
.whereRef('tag_asset.assetId', '=', 'asset.id'),
|
||||
).as('tags'),
|
||||
)
|
||||
.whereRef('asset_exif.assetId', '=', 'asset.id')
|
||||
.as('asset2'),
|
||||
(join) => join.onTrue(),
|
||||
)
|
||||
.select('asset.duplicateId')
|
||||
.select((eb) => eb.fn.jsonAgg('asset2').orderBy('asset.localDateTime', 'asc').$castTo<MapAsset[]>().as('assets'))
|
||||
.where('asset.duplicateId', '=', asUuid(duplicateId))
|
||||
.where('asset.deletedAt', 'is', null)
|
||||
.where('asset.stackId', 'is', null)
|
||||
.groupBy('asset.duplicateId')
|
||||
.executeTakeFirst();
|
||||
|
||||
if (!result || !result.duplicateId) {
|
||||
return;
|
||||
}
|
||||
|
||||
return { duplicateId: result.duplicateId, assets: result.assets };
|
||||
}
|
||||
|
||||
@GenerateSql({ params: [DummyValue.UUID, DummyValue.UUID] })
|
||||
async delete(userId: string, id: string): Promise<void> {
|
||||
await this.db
|
||||
@@ -134,7 +210,7 @@ export class DuplicateRepository {
|
||||
.where('asset.id', '!=', asUuid(assetId))
|
||||
.where('asset.stackId', 'is', null)
|
||||
.orderBy('distance')
|
||||
.limit(64),
|
||||
.limit(DUPLICATE_SEARCH_LIMIT),
|
||||
)
|
||||
.selectFrom('cte')
|
||||
.selectAll()
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
import { BulkIdErrorReason } from 'src/dtos/asset-ids.response.dto';
|
||||
import { MapAsset } from 'src/dtos/asset-response.dto';
|
||||
import { AssetType, AssetVisibility, JobName, JobStatus } from 'src/enum';
|
||||
import { DuplicateService } from 'src/services/duplicate.service';
|
||||
import { SearchService } from 'src/services/search.service';
|
||||
import { AssetFactory } from 'test/factories/asset.factory';
|
||||
import { authStub } from 'test/fixtures/auth.stub';
|
||||
import { getForDuplicate } from 'test/mappers';
|
||||
import { newUuid } from 'test/small.factory';
|
||||
import { makeStream, newTestService, ServiceMocks } from 'test/utils';
|
||||
import { beforeEach, vitest } from 'vitest';
|
||||
import { beforeEach, describe, expect, it, vitest } from 'vitest';
|
||||
|
||||
vitest.useFakeTimers();
|
||||
|
||||
@@ -26,7 +27,7 @@ const hasDupe = {
|
||||
duplicateId: 'duplicate-id',
|
||||
};
|
||||
|
||||
describe(SearchService.name, () => {
|
||||
describe(DuplicateService.name, () => {
|
||||
let sut: DuplicateService;
|
||||
let mocks: ServiceMocks;
|
||||
|
||||
@@ -41,6 +42,8 @@ describe(SearchService.name, () => {
|
||||
describe('getDuplicates', () => {
|
||||
it('should get duplicates', async () => {
|
||||
const asset = AssetFactory.from().exif().build();
|
||||
mocks.access.duplicate.checkOwnerAccess.mockResolvedValue(new Set(['duplicate-id']));
|
||||
mocks.duplicateRepository.cleanupSingletonGroups.mockResolvedValue();
|
||||
mocks.duplicateRepository.getAll.mockResolvedValue([
|
||||
{
|
||||
duplicateId: 'duplicate-id',
|
||||
@@ -51,9 +54,24 @@ describe(SearchService.name, () => {
|
||||
{
|
||||
duplicateId: 'duplicate-id',
|
||||
assets: [expect.objectContaining({ id: asset.id }), expect.objectContaining({ id: asset.id })],
|
||||
suggestedKeepAssetIds: [asset.id],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('should return suggestedKeepAssetIds based on file size', async () => {
|
||||
const smallAsset = AssetFactory.from().exif({ fileSizeInByte: 1000 }).build();
|
||||
const largeAsset = AssetFactory.from().exif({ fileSizeInByte: 5000 }).build();
|
||||
mocks.duplicateRepository.cleanupSingletonGroups.mockResolvedValue();
|
||||
mocks.duplicateRepository.getAll.mockResolvedValue([
|
||||
{
|
||||
duplicateId: 'duplicate-id',
|
||||
assets: [getForDuplicate(smallAsset), getForDuplicate(largeAsset)],
|
||||
},
|
||||
]);
|
||||
const result = await sut.getDuplicates(authStub.admin);
|
||||
expect(result[0].suggestedKeepAssetIds).toEqual([largeAsset.id]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('handleQueueSearchDuplicates', () => {
|
||||
@@ -131,6 +149,200 @@ describe(SearchService.name, () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolve', () => {
|
||||
it('should handle mixed success and failure', async () => {
|
||||
const asset = AssetFactory.create();
|
||||
mocks.access.duplicate.checkOwnerAccess.mockResolvedValue(new Set(['group-1', 'group-2']));
|
||||
mocks.duplicateRepository.get.mockResolvedValueOnce(void 0);
|
||||
mocks.duplicateRepository.get.mockResolvedValueOnce({
|
||||
duplicateId: 'group-2',
|
||||
assets: [asset as unknown as MapAsset],
|
||||
});
|
||||
|
||||
await expect(
|
||||
sut.resolve(authStub.admin, {
|
||||
groups: [
|
||||
{ duplicateId: 'group-1', keepAssetIds: [], trashAssetIds: [] },
|
||||
{ duplicateId: 'group-2', keepAssetIds: [asset.id], trashAssetIds: [] },
|
||||
],
|
||||
}),
|
||||
).resolves.toEqual([
|
||||
{ id: 'group-1', success: false, error: BulkIdErrorReason.NOT_FOUND },
|
||||
{ id: 'group-2', success: true },
|
||||
]);
|
||||
});
|
||||
|
||||
it('should catch and report errors', async () => {
|
||||
mocks.access.duplicate.checkOwnerAccess.mockResolvedValue(new Set(['group-1']));
|
||||
mocks.duplicateRepository.get.mockRejectedValue(new Error('Database error'));
|
||||
|
||||
await expect(
|
||||
sut.resolve(authStub.admin, {
|
||||
groups: [{ duplicateId: 'group-1', keepAssetIds: [], trashAssetIds: [] }],
|
||||
}),
|
||||
).resolves.toEqual([{ id: 'group-1', success: false, error: BulkIdErrorReason.UNKNOWN }]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveGroup (via resolve)', () => {
|
||||
it('should fail if duplicate group not found', async () => {
|
||||
mocks.access.duplicate.checkOwnerAccess.mockResolvedValue(new Set(['missing-id']));
|
||||
mocks.duplicateRepository.get.mockResolvedValue(void 0);
|
||||
|
||||
await expect(
|
||||
sut.resolve(authStub.admin, {
|
||||
groups: [{ duplicateId: 'missing-id', keepAssetIds: [], trashAssetIds: [] }],
|
||||
}),
|
||||
).resolves.toEqual([
|
||||
{
|
||||
id: 'missing-id',
|
||||
success: false,
|
||||
error: BulkIdErrorReason.NOT_FOUND,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('should skip when keepAssetIds contains non-member', async () => {
|
||||
const asset = AssetFactory.create();
|
||||
mocks.access.duplicate.checkOwnerAccess.mockResolvedValue(new Set(['group-1']));
|
||||
mocks.duplicateRepository.get.mockResolvedValue({
|
||||
duplicateId: 'group-1',
|
||||
assets: [asset as unknown as MapAsset],
|
||||
});
|
||||
|
||||
await expect(
|
||||
sut.resolve(authStub.admin, {
|
||||
groups: [{ duplicateId: 'group-1', keepAssetIds: ['asset-999', asset.id], trashAssetIds: [] }],
|
||||
}),
|
||||
).resolves.toEqual([{ id: 'group-1', success: true }]);
|
||||
});
|
||||
|
||||
it('should skip when trashAssetIds contains non-member', async () => {
|
||||
const asset = AssetFactory.create();
|
||||
mocks.access.duplicate.checkOwnerAccess.mockResolvedValue(new Set(['group-1']));
|
||||
mocks.duplicateRepository.get.mockResolvedValue({
|
||||
duplicateId: 'group-1',
|
||||
assets: [asset as unknown as MapAsset],
|
||||
});
|
||||
|
||||
await expect(
|
||||
sut.resolve(authStub.admin, {
|
||||
groups: [{ duplicateId: 'group-1', keepAssetIds: [asset.id], trashAssetIds: ['asset-999'] }],
|
||||
}),
|
||||
).resolves.toEqual([{ id: 'group-1', success: true }]);
|
||||
});
|
||||
|
||||
it('should fail if keepAssetIds and trashAssetIds overlap', async () => {
|
||||
const asset1 = AssetFactory.create();
|
||||
const asset2 = AssetFactory.create();
|
||||
mocks.access.duplicate.checkOwnerAccess.mockResolvedValue(new Set(['group-1']));
|
||||
mocks.duplicateRepository.get.mockResolvedValue({
|
||||
duplicateId: 'group-1',
|
||||
assets: [asset1 as unknown as MapAsset, asset2 as unknown as MapAsset],
|
||||
});
|
||||
|
||||
const result = await sut.resolve(authStub.admin, {
|
||||
groups: [{ duplicateId: 'group-1', keepAssetIds: [asset1.id], trashAssetIds: [asset1.id] }],
|
||||
});
|
||||
|
||||
expect(result[0].success).toBe(false);
|
||||
expect(result[0].errorMessage).toContain('An asset cannot be in both keepAssetIds and trashAssetIds');
|
||||
});
|
||||
|
||||
it('should fail if keepAssetIds and trashAssetIds do not cover all assets', async () => {
|
||||
const asset1 = AssetFactory.create();
|
||||
const asset2 = AssetFactory.create();
|
||||
const asset3 = AssetFactory.create();
|
||||
mocks.access.duplicate.checkOwnerAccess.mockResolvedValue(new Set(['group-1']));
|
||||
mocks.duplicateRepository.get.mockResolvedValue({
|
||||
duplicateId: 'group-1',
|
||||
assets: [asset1 as unknown as MapAsset, asset2 as unknown as MapAsset, asset3 as unknown as MapAsset],
|
||||
});
|
||||
|
||||
const result = await sut.resolve(authStub.admin, {
|
||||
groups: [{ duplicateId: 'group-1', keepAssetIds: [asset1.id], trashAssetIds: [asset2.id] }],
|
||||
});
|
||||
|
||||
expect(result[0].success).toBe(false);
|
||||
expect(result[0].errorMessage).toContain('Every asset must be in either keepAssetIds or trashAssetIds');
|
||||
});
|
||||
|
||||
it('should fail if partial trash without keepers', async () => {
|
||||
const asset1 = AssetFactory.create();
|
||||
const asset2 = AssetFactory.create();
|
||||
mocks.access.duplicate.checkOwnerAccess.mockResolvedValue(new Set(['group-1']));
|
||||
mocks.duplicateRepository.get.mockResolvedValue({
|
||||
duplicateId: 'group-1',
|
||||
assets: [asset1 as unknown as MapAsset, asset2 as unknown as MapAsset],
|
||||
});
|
||||
|
||||
const result = await sut.resolve(authStub.admin, {
|
||||
groups: [{ duplicateId: 'group-1', keepAssetIds: [], trashAssetIds: [asset1.id] }],
|
||||
});
|
||||
|
||||
expect(result[0].success).toBe(false);
|
||||
expect(result[0].errorMessage).toContain('Every asset must be in either keepAssetIds or trashAssetIds');
|
||||
});
|
||||
|
||||
it('should sync merged tags to asset_exif.tags', async () => {
|
||||
const asset1 = AssetFactory.create();
|
||||
const asset2 = AssetFactory.create();
|
||||
mocks.access.duplicate.checkOwnerAccess.mockResolvedValue(new Set(['group-1']));
|
||||
mocks.access.asset.checkOwnerAccess.mockResolvedValue(new Set(['asset-2']));
|
||||
mocks.access.tag.checkOwnerAccess.mockResolvedValue(new Set(['tag-1', 'tag-2']));
|
||||
mocks.duplicateRepository.get.mockResolvedValue({
|
||||
duplicateId: 'group-1',
|
||||
assets: [
|
||||
{
|
||||
...asset1,
|
||||
tags: [
|
||||
{
|
||||
id: 'tag-1',
|
||||
value: 'Work',
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
parentId: null,
|
||||
color: null,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
...asset2,
|
||||
tags: [
|
||||
{
|
||||
id: 'tag-2',
|
||||
value: 'Travel',
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
parentId: null,
|
||||
color: null,
|
||||
},
|
||||
],
|
||||
},
|
||||
] as any,
|
||||
});
|
||||
|
||||
const result = await sut.resolve(authStub.admin, {
|
||||
groups: [{ duplicateId: 'group-1', keepAssetIds: [asset1.id], trashAssetIds: [asset2.id] }],
|
||||
});
|
||||
|
||||
expect(result[0].success).toBe(true);
|
||||
|
||||
// Verify tags were applied to tag_asset table
|
||||
expect(mocks.tag.replaceAssetTags).toHaveBeenCalledWith(asset1.id, ['tag-1', 'tag-2']);
|
||||
|
||||
// Verify merged tag values were written to asset_exif.tags so SidecarWrite preserves them
|
||||
expect(mocks.asset.updateAllExif).toHaveBeenCalledWith([asset1.id], { tags: ['Work', 'Travel'] });
|
||||
|
||||
// Verify SidecarWrite was queued (to write tags to sidecar)
|
||||
expect(mocks.job.queueAll).toHaveBeenCalledWith([{ name: JobName.SidecarWrite, data: { id: asset1.id } }]);
|
||||
});
|
||||
|
||||
// NOTE: The following integration-style tests are covered by E2E tests instead
|
||||
// to avoid complex mock setup. The validation and error-handling logic above
|
||||
// is thoroughly unit tested.
|
||||
});
|
||||
|
||||
describe('handleSearchDuplicates', () => {
|
||||
beforeEach(() => {
|
||||
mocks.systemMetadata.get.mockResolvedValue({
|
||||
|
||||
@@ -1,24 +1,84 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
import { JOBS_ASSET_PAGINATION_SIZE } from 'src/constants';
|
||||
import { OnJob } from 'src/decorators';
|
||||
import { BulkIdsDto } from 'src/dtos/asset-ids.response.dto';
|
||||
import { mapAsset } from 'src/dtos/asset-response.dto';
|
||||
import { BulkIdErrorReason, BulkIdResponseDto, BulkIdsDto } from 'src/dtos/asset-ids.response.dto';
|
||||
import { MapAsset, mapAsset } from 'src/dtos/asset-response.dto';
|
||||
import { AuthDto } from 'src/dtos/auth.dto';
|
||||
import { DuplicateResponseDto } from 'src/dtos/duplicate.dto';
|
||||
import { AssetVisibility, JobName, JobStatus, QueueName } from 'src/enum';
|
||||
import { DuplicateResolveDto, DuplicateResolveGroupDto, DuplicateResponseDto } from 'src/dtos/duplicate.dto';
|
||||
import { AssetStatus, AssetVisibility, JobName, JobStatus, Permission, QueueName } from 'src/enum';
|
||||
import { AssetDuplicateResult } from 'src/repositories/search.repository';
|
||||
import { BaseService } from 'src/services/base.service';
|
||||
import { JobItem, JobOf } from 'src/types';
|
||||
import { suggestDuplicateKeepAssetIds } from 'src/utils/duplicate';
|
||||
import { isDuplicateDetectionEnabled } from 'src/utils/misc';
|
||||
|
||||
type ResolveRequest = {
|
||||
assetUpdate: {
|
||||
isFavorite?: boolean;
|
||||
visibility?: AssetVisibility;
|
||||
};
|
||||
|
||||
exifUpdate: {
|
||||
rating?: number;
|
||||
latitude?: number;
|
||||
longitude?: number;
|
||||
description?: string;
|
||||
};
|
||||
|
||||
mergedAlbumIds: string[];
|
||||
|
||||
mergedTagIds: string[];
|
||||
|
||||
mergedTagValues: string[];
|
||||
};
|
||||
|
||||
const uniqueNonEmptyLines = (values: Array<string | null | undefined>): string[] => {
|
||||
const unique = new Set<string>();
|
||||
const lines: string[] = [];
|
||||
for (const value of values) {
|
||||
if (!value) {
|
||||
continue;
|
||||
}
|
||||
for (const line of value.split(/\r?\n/)) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed || unique.has(trimmed)) {
|
||||
continue;
|
||||
}
|
||||
unique.add(trimmed);
|
||||
lines.push(trimmed);
|
||||
}
|
||||
}
|
||||
return lines;
|
||||
};
|
||||
|
||||
const getUniqueCoordinate = (assets: MapAsset[], key: 'latitude' | 'longitude'): number | null => {
|
||||
const values = assets
|
||||
.map((asset) => asset.exifInfo?.[key])
|
||||
.filter((value): value is number => Number.isFinite(value));
|
||||
|
||||
if (values.length === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const unique = new Set(values);
|
||||
return unique.size === 1 ? [...unique][0] : null;
|
||||
};
|
||||
|
||||
@Injectable()
|
||||
export class DuplicateService extends BaseService {
|
||||
async getDuplicates(auth: AuthDto): Promise<DuplicateResponseDto[]> {
|
||||
// Clean up singleton groups (assets that are the only member of their duplicate group)
|
||||
await this.duplicateRepository.cleanupSingletonGroups(auth.user.id);
|
||||
|
||||
const duplicates = await this.duplicateRepository.getAll(auth.user.id);
|
||||
return duplicates.map(({ duplicateId, assets }) => ({
|
||||
duplicateId,
|
||||
assets: assets.map((asset) => mapAsset(asset, { auth })),
|
||||
}));
|
||||
return duplicates.map(({ duplicateId, assets }) => {
|
||||
const mappedAssets = assets.map((asset) => mapAsset(asset, { auth }));
|
||||
return {
|
||||
duplicateId,
|
||||
assets: mappedAssets,
|
||||
suggestedKeepAssetIds: suggestDuplicateKeepAssetIds(mappedAssets),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
async delete(auth: AuthDto, id: string): Promise<void> {
|
||||
@@ -29,6 +89,213 @@ export class DuplicateService extends BaseService {
|
||||
await this.duplicateRepository.deleteAll(auth.user.id, dto.ids);
|
||||
}
|
||||
|
||||
async resolve(auth: AuthDto, dto: DuplicateResolveDto) {
|
||||
const duplicateIds = dto.groups.map(({ duplicateId }) => duplicateId);
|
||||
|
||||
await this.requireAccess({ auth, permission: Permission.DuplicateDelete, ids: duplicateIds });
|
||||
|
||||
const results: BulkIdResponseDto[] = [];
|
||||
|
||||
for (const group of dto.groups) {
|
||||
try {
|
||||
results.push(await this.resolveGroup(auth, group));
|
||||
} catch (error: Error | any) {
|
||||
this.logger.error(`Error resolving duplicate group ${group.duplicateId}: ${error}`, error?.stack);
|
||||
results.push({ id: group.duplicateId, success: false, error: BulkIdErrorReason.UNKNOWN });
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
private async resolveGroup(auth: AuthDto, group: DuplicateResolveGroupDto): Promise<BulkIdResponseDto> {
|
||||
const { duplicateId, keepAssetIds, trashAssetIds } = group;
|
||||
|
||||
const duplicateGroup = await this.duplicateRepository.get(duplicateId);
|
||||
if (!duplicateGroup) {
|
||||
return { id: duplicateId, success: false, error: BulkIdErrorReason.NOT_FOUND };
|
||||
}
|
||||
|
||||
const groupAssetIds = new Set(duplicateGroup.assets.map((a) => a.id));
|
||||
|
||||
// ignore/skip asset IDs not in the group
|
||||
const idsToKeep = keepAssetIds.filter((id) => groupAssetIds.has(id));
|
||||
const idsToTrash = trashAssetIds.filter((id) => groupAssetIds.has(id));
|
||||
|
||||
for (const assetId of groupAssetIds) {
|
||||
if (idsToKeep.includes(assetId) && idsToTrash.includes(assetId)) {
|
||||
return {
|
||||
id: duplicateId,
|
||||
success: false,
|
||||
error: BulkIdErrorReason.VALIDATION,
|
||||
errorMessage: 'An asset cannot be in both keepAssetIds and trashAssetIds',
|
||||
};
|
||||
}
|
||||
|
||||
if (!idsToKeep.includes(assetId) && !idsToTrash.includes(assetId)) {
|
||||
return {
|
||||
id: duplicateId,
|
||||
success: false,
|
||||
error: BulkIdErrorReason.VALIDATION,
|
||||
errorMessage: 'Every asset must be in either keepAssetIds or trashAssetIds',
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
if (idsToTrash.length > 0) {
|
||||
const ids = await this.checkAccess({ auth, permission: Permission.AssetDelete, ids: idsToTrash });
|
||||
if (ids.size !== idsToTrash.length) {
|
||||
return {
|
||||
id: duplicateId,
|
||||
success: false,
|
||||
error: BulkIdErrorReason.NO_PERMISSION,
|
||||
errorMessage: 'No permission to delete assets',
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
const assetAlbumMap = await this.albumRepository.getByAssetIds(auth.user.id, [...groupAssetIds]);
|
||||
|
||||
const { assetUpdate, exifUpdate, mergedAlbumIds, mergedTagIds, mergedTagValues } = this.getSyncMergeResult(
|
||||
duplicateGroup.assets,
|
||||
assetAlbumMap,
|
||||
);
|
||||
|
||||
if (mergedAlbumIds.length > 0) {
|
||||
const allowedAlbumIds = await this.checkAccess({
|
||||
auth,
|
||||
permission: Permission.AlbumAssetCreate,
|
||||
ids: mergedAlbumIds,
|
||||
});
|
||||
|
||||
const allowedShareIds = await this.checkAccess({
|
||||
auth,
|
||||
permission: Permission.AssetShare,
|
||||
ids: idsToKeep,
|
||||
});
|
||||
|
||||
if (allowedAlbumIds.size > 0 && allowedShareIds.size > 0) {
|
||||
await this.albumRepository.addAssetIdsToAlbums(
|
||||
[...allowedAlbumIds].flatMap((albumId) => [...allowedShareIds].map((assetId) => ({ albumId, assetId }))),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (mergedTagIds.length > 0) {
|
||||
const allowedTagIds = await this.checkAccess({
|
||||
auth,
|
||||
permission: Permission.TagAsset,
|
||||
ids: mergedTagIds,
|
||||
});
|
||||
|
||||
if (allowedTagIds.size > 0) {
|
||||
// Replace tags for each keeper asset to ensure all merged tags are applied
|
||||
await Promise.all(idsToKeep.map((assetId) => this.tagRepository.replaceAssetTags(assetId, [...allowedTagIds])));
|
||||
|
||||
// Update asset_exif.tags so the subsequent SidecarWrite + MetadataExtraction
|
||||
// cycle preserves the merged tags (updateAllExif locks the property automatically)
|
||||
await this.assetRepository.updateAllExif(idsToKeep, { tags: mergedTagValues });
|
||||
}
|
||||
}
|
||||
|
||||
if (idsToKeep.length > 0) {
|
||||
const hasExifUpdate = Object.keys(exifUpdate).length > 0;
|
||||
const hasTagUpdate = mergedTagIds.length > 0;
|
||||
|
||||
if (hasExifUpdate) {
|
||||
await this.assetRepository.updateAllExif(idsToKeep, exifUpdate);
|
||||
}
|
||||
|
||||
if (hasExifUpdate || hasTagUpdate) {
|
||||
await this.jobRepository.queueAll(idsToKeep.map((id) => ({ name: JobName.SidecarWrite, data: { id } })));
|
||||
}
|
||||
|
||||
await this.assetRepository.updateAll(idsToKeep, { duplicateId: null, ...assetUpdate });
|
||||
}
|
||||
|
||||
if (idsToTrash.length > 0) {
|
||||
// TODO: this is duplicated with AssetService.deleteAssets
|
||||
const { trash } = await this.getConfig({ withCache: true });
|
||||
const force = !trash.enabled;
|
||||
|
||||
await this.assetRepository.updateAll(idsToTrash, {
|
||||
deletedAt: new Date(),
|
||||
status: force ? AssetStatus.Deleted : AssetStatus.Trashed,
|
||||
duplicateId: null,
|
||||
});
|
||||
|
||||
await this.eventRepository.emit(force ? 'AssetDeleteAll' : 'AssetTrashAll', {
|
||||
assetIds: idsToTrash,
|
||||
userId: auth.user.id,
|
||||
});
|
||||
}
|
||||
|
||||
return { id: duplicateId, success: true };
|
||||
}
|
||||
|
||||
private getSyncMergeResult(assets: MapAsset[], assetAlbumMap: Map<string, string[]> = new Map()): ResolveRequest {
|
||||
const response: ResolveRequest = {
|
||||
mergedAlbumIds: [],
|
||||
mergedTagIds: [],
|
||||
mergedTagValues: [],
|
||||
assetUpdate: {},
|
||||
exifUpdate: {},
|
||||
};
|
||||
|
||||
response.assetUpdate.isFavorite = assets.some((asset) => asset.isFavorite);
|
||||
|
||||
const visibilityOrder = [AssetVisibility.Locked, AssetVisibility.Archive, AssetVisibility.Timeline];
|
||||
let visibility = visibilityOrder.find((level) => assets.some((asset) => asset.visibility === level));
|
||||
if (!visibility && assets.some((asset) => asset.visibility === AssetVisibility.Hidden)) {
|
||||
visibility = AssetVisibility.Hidden;
|
||||
}
|
||||
if (visibility) {
|
||||
response.assetUpdate.visibility = visibility;
|
||||
}
|
||||
|
||||
let rating = 0;
|
||||
for (const asset of assets) {
|
||||
const assetRating = asset.exifInfo?.rating ?? 0;
|
||||
if (assetRating > rating) {
|
||||
rating = assetRating;
|
||||
}
|
||||
}
|
||||
if (rating > 0) {
|
||||
response.exifUpdate.rating = rating;
|
||||
}
|
||||
|
||||
const descriptionLines = uniqueNonEmptyLines(assets.map((asset) => asset.exifInfo?.description));
|
||||
const description = descriptionLines.length > 0 ? descriptionLines.join('\n') : null;
|
||||
if (description !== null) {
|
||||
response.exifUpdate.description = description;
|
||||
}
|
||||
|
||||
const latitude = getUniqueCoordinate(assets, 'latitude');
|
||||
const longitude = getUniqueCoordinate(assets, 'longitude');
|
||||
if (latitude !== null && longitude !== null) {
|
||||
response.exifUpdate.latitude = latitude;
|
||||
response.exifUpdate.longitude = longitude;
|
||||
}
|
||||
|
||||
const albumIdSet = new Set<string>();
|
||||
for (const [, albumIds] of assetAlbumMap) {
|
||||
for (const albumId of albumIds) {
|
||||
albumIdSet.add(albumId);
|
||||
}
|
||||
}
|
||||
response.mergedAlbumIds = [...albumIdSet];
|
||||
|
||||
const allTags = assets.flatMap((asset) => asset.tags ?? []);
|
||||
const tagIds = [...new Set(allTags.map((tag) => tag.id).filter((id): id is string => !!id))];
|
||||
const tagValues = [...new Set(allTags.map((tag) => tag.value).filter((v): v is string => !!v))];
|
||||
if (tagIds.length > 0) {
|
||||
response.mergedTagIds = tagIds;
|
||||
response.mergedTagValues = tagValues;
|
||||
}
|
||||
|
||||
return response;
|
||||
}
|
||||
|
||||
@OnJob({ name: JobName.AssetDetectDuplicatesQueueAll, queue: QueueName.DuplicateDetection })
|
||||
async handleQueueSearchDuplicates({ force }: JobOf<JobName.AssetDetectDuplicatesQueueAll>): Promise<JobStatus> {
|
||||
const { machineLearning } = await this.getConfig({ withCache: false });
|
||||
|
||||
@@ -241,6 +241,11 @@ const checkOtherAccess = async (access: AccessRepository, request: OtherAccessRe
|
||||
return ids.has(auth.user.id) ? new Set([auth.user.id]) : new Set();
|
||||
}
|
||||
|
||||
case Permission.DuplicateRead:
|
||||
case Permission.DuplicateDelete: {
|
||||
return access.duplicate.checkOwnerAccess(auth.user.id, ids);
|
||||
}
|
||||
|
||||
case Permission.AuthDeviceDelete: {
|
||||
return await access.authDevice.checkOwnerAccess(auth.user.id, ids);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
import { AssetResponseDto } from 'src/dtos/asset-response.dto';
|
||||
import { AssetType, AssetVisibility } from 'src/enum';
|
||||
import { getExifCount, suggestDuplicate, suggestDuplicateKeepAssetIds } from 'src/utils/duplicate';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
const createAsset = (
|
||||
id: string,
|
||||
fileSizeInByte: number | null = null,
|
||||
exifFields: Record<string, unknown> = {},
|
||||
): AssetResponseDto => ({
|
||||
id,
|
||||
type: AssetType.Image,
|
||||
thumbhash: null,
|
||||
localDateTime: new Date().toISOString(),
|
||||
duration: '0:00:00.00000',
|
||||
hasMetadata: true,
|
||||
width: 1920,
|
||||
height: 1080,
|
||||
createdAt: new Date().toISOString(),
|
||||
deviceAssetId: 'device-asset-1',
|
||||
deviceId: 'device-1',
|
||||
ownerId: 'owner-1',
|
||||
originalPath: '/path/to/asset',
|
||||
originalFileName: 'asset.jpg',
|
||||
fileCreatedAt: new Date().toISOString(),
|
||||
fileModifiedAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
isFavorite: false,
|
||||
isArchived: false,
|
||||
isTrashed: false,
|
||||
isOffline: false,
|
||||
isEdited: false,
|
||||
visibility: AssetVisibility.Timeline,
|
||||
checksum: 'checksum',
|
||||
exifInfo:
|
||||
fileSizeInByte !== null || Object.keys(exifFields).length > 0 ? { fileSizeInByte, ...exifFields } : undefined,
|
||||
});
|
||||
|
||||
describe('duplicate utils', () => {
|
||||
describe('getExifCount', () => {
|
||||
it('should return 0 for asset without exifInfo', () => {
|
||||
const asset = createAsset('asset-1');
|
||||
asset.exifInfo = undefined;
|
||||
expect(getExifCount(asset)).toBe(0);
|
||||
});
|
||||
|
||||
it('should return 0 for empty exifInfo', () => {
|
||||
const asset = createAsset('asset-1');
|
||||
asset.exifInfo = {};
|
||||
expect(getExifCount(asset)).toBe(0);
|
||||
});
|
||||
|
||||
it('should count all truthy values in exifInfo', () => {
|
||||
const asset = createAsset('asset-1', 1000, {
|
||||
make: 'Canon',
|
||||
model: 'EOS 5D',
|
||||
dateTimeOriginal: new Date(),
|
||||
timeZone: 'UTC',
|
||||
latitude: 40.7128,
|
||||
longitude: -74.006,
|
||||
city: 'New York',
|
||||
state: 'NY',
|
||||
country: 'USA',
|
||||
description: 'A photo',
|
||||
rating: 5,
|
||||
});
|
||||
// fileSizeInByte (1000) + 11 other truthy fields = 12
|
||||
expect(getExifCount(asset)).toBe(12);
|
||||
});
|
||||
|
||||
it('should not count null or undefined values', () => {
|
||||
const asset = createAsset('asset-1', 1000, {
|
||||
make: 'Canon',
|
||||
model: null,
|
||||
latitude: undefined,
|
||||
city: '',
|
||||
rating: 0,
|
||||
});
|
||||
// fileSizeInByte (1000) + make ('Canon') = 2 truthy values
|
||||
// model (null), latitude (undefined), city (''), rating (0) are all falsy
|
||||
expect(getExifCount(asset)).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('suggestDuplicate', () => {
|
||||
it('should return undefined for empty list', () => {
|
||||
expect(suggestDuplicate([])).toBeUndefined();
|
||||
});
|
||||
|
||||
it('should return the single asset for list with one asset', () => {
|
||||
const asset = createAsset('asset-1', 1000);
|
||||
expect(suggestDuplicate([asset])).toEqual(asset);
|
||||
});
|
||||
|
||||
it('should return asset with largest file size', () => {
|
||||
const small = createAsset('small', 1000);
|
||||
const large = createAsset('large', 5000);
|
||||
const medium = createAsset('medium', 3000);
|
||||
|
||||
expect(suggestDuplicate([small, large, medium])?.id).toBe('large');
|
||||
expect(suggestDuplicate([large, small, medium])?.id).toBe('large');
|
||||
expect(suggestDuplicate([medium, small, large])?.id).toBe('large');
|
||||
});
|
||||
|
||||
it('should use EXIF count as tie-breaker when file sizes are equal', () => {
|
||||
const lessExif = createAsset('less-exif', 1000, { make: 'Canon' });
|
||||
const moreExif = createAsset('more-exif', 1000, {
|
||||
make: 'Canon',
|
||||
model: 'EOS 5D',
|
||||
dateTimeOriginal: new Date(),
|
||||
city: 'New York',
|
||||
});
|
||||
|
||||
expect(suggestDuplicate([lessExif, moreExif])?.id).toBe('more-exif');
|
||||
expect(suggestDuplicate([moreExif, lessExif])?.id).toBe('more-exif');
|
||||
});
|
||||
|
||||
it('should handle assets with no exifInfo (treat as 0 file size)', () => {
|
||||
const noExif = createAsset('no-exif');
|
||||
noExif.exifInfo = undefined;
|
||||
const withExif = createAsset('with-exif', 1000);
|
||||
|
||||
expect(suggestDuplicate([noExif, withExif])?.id).toBe('with-exif');
|
||||
});
|
||||
|
||||
it('should handle assets with exifInfo but no fileSizeInByte', () => {
|
||||
const noFileSize = createAsset('no-file-size');
|
||||
noFileSize.exifInfo = { make: 'Canon', model: 'EOS 5D' };
|
||||
const withFileSize = createAsset('with-file-size', 1000);
|
||||
|
||||
expect(suggestDuplicate([noFileSize, withFileSize])?.id).toBe('with-file-size');
|
||||
});
|
||||
|
||||
it('should return last asset when all have same file size and EXIF count', () => {
|
||||
const asset1 = createAsset('asset-1', 1000, { make: 'Canon' });
|
||||
const asset2 = createAsset('asset-2', 1000, { make: 'Nikon' });
|
||||
|
||||
// Both have same file size (1000) and same EXIF count (2: fileSizeInByte + make)
|
||||
// Should return the last one in the sorted array
|
||||
const result = suggestDuplicate([asset1, asset2]);
|
||||
// Since they're equal, the last one after sorting should be returned
|
||||
expect(result).toBeDefined();
|
||||
expect(['asset-1', 'asset-2']).toContain(result?.id);
|
||||
});
|
||||
|
||||
it('should prioritize file size over EXIF count', () => {
|
||||
const largeWithLessExif = createAsset('large-less-exif', 5000, { make: 'Canon' });
|
||||
const smallWithMoreExif = createAsset('small-more-exif', 1000, {
|
||||
make: 'Canon',
|
||||
model: 'EOS 5D',
|
||||
dateTimeOriginal: new Date(),
|
||||
city: 'New York',
|
||||
state: 'NY',
|
||||
country: 'USA',
|
||||
});
|
||||
|
||||
expect(suggestDuplicate([largeWithLessExif, smallWithMoreExif])?.id).toBe('large-less-exif');
|
||||
});
|
||||
});
|
||||
|
||||
describe('suggestDuplicateKeepAssetIds', () => {
|
||||
it('should return empty array for empty list', () => {
|
||||
expect(suggestDuplicateKeepAssetIds([])).toEqual([]);
|
||||
});
|
||||
|
||||
it('should return array with single asset ID', () => {
|
||||
const asset = createAsset('asset-1', 1000);
|
||||
expect(suggestDuplicateKeepAssetIds([asset])).toEqual(['asset-1']);
|
||||
});
|
||||
|
||||
it('should return array with best asset ID', () => {
|
||||
const small = createAsset('small', 1000);
|
||||
const large = createAsset('large', 5000);
|
||||
|
||||
expect(suggestDuplicateKeepAssetIds([small, large])).toEqual(['large']);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,60 @@
|
||||
import { AssetResponseDto } from 'src/dtos/asset-response.dto';
|
||||
|
||||
/**
|
||||
* Counts all truthy values in the exifInfo object.
|
||||
* This matches the client implementation in web/src/lib/utils/exif-utils.ts
|
||||
*
|
||||
* @param asset Asset with optional exifInfo
|
||||
* @returns Count of truthy EXIF values
|
||||
*/
|
||||
export const getExifCount = (asset: AssetResponseDto): number => {
|
||||
return Object.values(asset.exifInfo ?? {}).filter(Boolean).length;
|
||||
};
|
||||
|
||||
/**
|
||||
* Suggests the best duplicate asset to keep from a list of duplicates.
|
||||
* This is a direct port of the client logic from web/src/lib/utils/duplicate-utils.ts
|
||||
*
|
||||
* The best asset is determined by the following criteria:
|
||||
* 1. Largest image file size in bytes
|
||||
* 2. Largest count of EXIF data (as tie-breaker)
|
||||
*
|
||||
* @param assets List of duplicate assets
|
||||
* @returns The best asset to keep, or undefined if empty list
|
||||
*/
|
||||
export const suggestDuplicate = (assets: AssetResponseDto[]): AssetResponseDto | undefined => {
|
||||
if (assets.length === 0) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
// Sort by file size ascending (smallest first)
|
||||
let duplicateAssets = [...assets].toSorted(
|
||||
(a, b) => (a.exifInfo?.fileSizeInByte ?? 0) - (b.exifInfo?.fileSizeInByte ?? 0),
|
||||
);
|
||||
|
||||
// Get the largest file size (last element after sorting)
|
||||
const largestFileSize = duplicateAssets.at(-1)?.exifInfo?.fileSizeInByte ?? 0;
|
||||
|
||||
// Filter to keep only assets with the largest file size
|
||||
duplicateAssets = duplicateAssets.filter((asset) => (asset.exifInfo?.fileSizeInByte ?? 0) === largestFileSize);
|
||||
|
||||
// If there are multiple assets with the same file size, sort by EXIF count
|
||||
if (duplicateAssets.length >= 2) {
|
||||
duplicateAssets = duplicateAssets.toSorted((a, b) => getExifCount(a) - getExifCount(b));
|
||||
}
|
||||
|
||||
// Return the last asset (highest EXIF count among highest file size)
|
||||
return duplicateAssets.at(-1);
|
||||
};
|
||||
|
||||
/**
|
||||
* Suggests the best duplicate asset IDs to keep from a list of duplicates.
|
||||
* Returns an array with a single asset ID (the best candidate), or empty if no assets.
|
||||
*
|
||||
* @param assets List of duplicate assets
|
||||
* @returns Array of suggested asset IDs to keep (0 or 1 element)
|
||||
*/
|
||||
export const suggestDuplicateKeepAssetIds = (assets: AssetResponseDto[]): string[] => {
|
||||
const suggested = suggestDuplicate(assets);
|
||||
return suggested ? [suggested.id] : [];
|
||||
};
|
||||
Reference in New Issue
Block a user