Skip to content

IAV-86/feat: Implement comprehensive file manager service with batch operations and advanced transaction management

salaheddine zidani a demandé de fusionner feature/IAV-86 vers develop

Description

This MR introduces the iavia-file-manager-service - a complete file management microservice with AWS S3 storage, advanced transaction management, and comprehensive audit capabilities.

Key Features

🚀 Core Functionality

  • Multi-file upload with batch support and validation (extension, size, MIME type)
  • Agent Family KB specialized upload endpoint
  • File download with streaming and presigned/public URL generation
  • Metadata management (single, batch, existence check)
  • Smart deletion (single with rollback, bulk with detailed tracking)
  • Search & listing (by usage type, uploader, folder, name)
  • Comprehensive audit with FileAccessLog for all operations

🔒 Transaction Management (2-Phase)

  • Upload: S3 first → DB save → Auto cleanup orphaned files on failure
  • Delete: DB delete → S3 delete → Auto rollback on S3 failure
  • Bulk delete: Track fully deleted, DB-only deleted (️), and failed files

📊 Result DTOs

  • FileUploadResultDTO: Success/failure breakdown
  • BulkDeleteResultDTO: Three categories (fully deleted, DB-only, failed) + failure reasons
  • PresignedUrlDTO: Temporary secure access
  • FileDownloadDTO: Download info with Resource

Code Quality

Refactoring

  • **~300 lines eliminated through centralized helpers
  • 2 key helper methods: getFileMetadataOrThrow, getS3KeyFromMetadata
  • Enhanced S3 cleanup: Continues even if individual deletions fail
  • Proper transactions: @Transactional with REQUIRES_NEW for audit logs

Documentation

  1. FILE_MANAGER_TECHNICAL_DOCUMENTATION.md: Complete service architecture and patterns
  2. FILE_MANAGER_EXTERNAL_INTEGRATION_GUIDE.md: Step-by-step integration for external services

Configuration (Examples of non-real values for properties Table in config-server)

Application Profile Label Key Value
iavia-agent-service-customer-care local develop aws.credentials.accessKey ABCDEFGH
iavia-agent-service-customer-care local develop aws.credentials.secretKey aZ7E8r9t21Y
iavia-agent-service-customer-care local develop aws.bucketName iavia-customer-care-dev
iavia-agent-service-customer-care local develop aws.bucketPolicy classpath:bucketPolicy.json
iavia-agent-service-customer-care local develop aws.preSignedUrlExpiration.defaultTTL 300
iavia-agent-service-customer-care local develop aws.lifecycle.documentsTTL 30
iavia-agent-service-customer-care local develop aws.lifecycle.imagesTTL 90
iavia-agent-service-customer-care local develop aws.lifecycle.videosTTL 60
iavia-agent-service-customer-care local develop aws.max-file-size.images 10MB
iavia-agent-service-customer-care local develop aws.max-file-size.videos 100MB
iavia-agent-service-customer-care local develop aws.max-file-size.documents 15MB
iavia-agent-service-customer-care local develop aws.max-file-size.default-size 5MB
iavia-agent-service-customer-care local develop aws.expiration-action.images delete
iavia-agent-service-customer-care local develop aws.expiration-action.videos archive
iavia-agent-service-customer-care local develop aws.expiration-action.documents manual_review
iavia-agent-service-customer-care local develop aws.allowedExtensions "jpg,jpeg,png,pdf,docx"
iavia-agent-service-customer-care local develop aws.pagination.pageSize 20

Testing

  • Upload (single, batch, with failures)
  • Download and URL generation
  • Metadata operations
  • Delete (single with rollback, bulk with partial failures)
  • Search and listing
  • Access log recording

Important Notes

Orphaned S3 files: Manual cleanup required when dbOnlyDeletedUids is not empty No auto-retry: Handle client-side if needed

Closes IAV-86

Modification effectuée par salaheddine zidani

Rapports de requête de fusion