Archives Video Appraiser (AVA) – Poster Presentation

The World Bank Group (WBG), founded in 1945, is the oldest and largest multilateral development bank in the world.  It is one of the largest sources of funding and knowledge for developing countries; a unique global partnership of five institutions dedicated to ending extreme poverty, increasing shared prosperity, and promoting sustainable development. With 189 member countries and more than 12,000 staff worldwide, the WBG works with public and private sector partners, investing in groundbreaking projects and using data, research, and technology to develop solutions to the most urgent global challenges.  

The WBG Archives provides the public with access to the archival holdings of the WBG along with engaging tools that enable the discovery of historical information. It also has fiduciary responsibilities for current records management and information governance within the WBG, made possible by policies, programs and services provided internally.

One of the responsibilities of the WBG Archives is to appraise the business, legal and research value of WBG records to identify records with permanent value and records that can be destroyed when no longer needed. The ever-expanding volume of uncategorized digital records makes the manual appraisal and selection of records increasingly labor-intensive and prone to mistakes. In 2021, WBG archivists and their IT counterparts developed a prototype to test the effectiveness of using Machine Learning (ML) technology to assist the Archives in the appraisal and selection of born-digital moving image records.

The poster introduces the results of this case study: the creation of the so-called Archival Video Appraiser (AVA), a ML tool that generates recommendations on which video recordings should be permanently preserved or destroyed based on a set of pre-defined appraisal criteria. Furthermore, AVA also automatically transfers the records to a designated folder based on the decisions.

Problem Statement

The exponential growth of digital content makes it hard, if not impossible, for the WBG Archives to appraise the vast volume of unclassified records that we receive into custody. An example are video recordings, of which we receive an average of 200 per month. Many of these recordings have permanent value according to our Retention Policies, while others need to be destroyed. Some recordings are transferred to our custody with promising and descriptive titles, but contain no sound or content (e.g., a meeting scheduled to be automatically recorded via Webex is cancelled and recorded anyway).

The process of human-driven appraisal of recordings requires the visual review of a one-minute segment of every video to ensure that there’s content and sound and to determine which retention rule applies. Archivists also use metadata, such as the title or the date of the recording, to support the decisions. Once decisions are documented, the archivist manually transfers the videos to different locations depending on whether they are eligible for ingest into the WBG’s digital preservation platform (internally named Digital Vault) or are ready to be destroyed.

The process takes about 2-person days per month and is prone to errors due to its very manual nature.

Project Goal

Project Process

The effort to create AVA required extensive collaboration between the WBG archivists and the IT staff assigned to the project. Initially, there needed to be knowledge exchange as the archivists learned about the possible ways to use ML and the IT team learned about how the appraisal and selection process for born digital moving images was being performed manually. The actual implementation was an iterative process involving training on an initial set of 1000 pre-appraised videos and subsequent testing of results against manual decisions. It was during these efforts that we discovered that we needed more sample videos that had already been through the manual appraisal and selection process, forcing us to increase the training set to 3000 videos.

Technology Used

M365: PowerApps, PowerApps API and Power Automate (Flow).

AWS: API Service​, Python, Flask, Celery​, ECS Lambda​, Rekognition and RabbitMQ.

Technology Drill Down

Web App:

• View list of existing video, and decisions (recommended and final)
• Confirm the AI/ML decision, or decide to keep/delete
• Based on user request, issue commands to remote system to: 
            
  • Move Delete videos to ‘Delete folder’
  • Move Keep videos to ‘Archive folder’
  • Scan for new videos
  • Update status of tasks
  • Generate frames for video

Remote Worker 1 (On Prem)

• Scan files system to create a list of  videos
• If new, upload to DB with id and path, and change name to name+id
• Purge specific videos
• Generate text from image
• Store text, images in S3
• Initiate Recognition on images if needed

AI Component

• C1 - Given a title, predict Keep/Delete (could be Comprehend)
• C2  - Generate text from image using Textract
• C3 – Get object labels for images using Rekognition
• C4 – Voice to text using Transcribe

CDS

• Store all the videos metadata, and associated decisions

S3

• Stores images, text and labels for videos

Results

• AVA scans designated network folders for new videos and stores the information in a database.
• AVA then extracts and analyzes the filename and a generated transcript, identifies empty and soundless recordings, and provides recommendations for archival retention or destruction.
• AVA’s preliminary decisions are validated by an archivist.
• Videos identified for archival retention are transferred to a digital preservation staging area and those identified for destruction are destroyed.
• Audit reports are generated and automatically captured in the WBG’s Electronic Document and Records Management System.

AVA’s accuracy currently averages 85%. AVA is also successful in detecting empty and soundless videos. The tool requires approximately 10 minutes to make the recommendations for 200 videos. Once the recommendations are available, an archivist needs about 30 minutes to verify and take any required corrective actions.

AVA’s implementation has resulted in 1.5-person day savings per month, a reduction of manual errors, and an increase in appraisal decision accuracy.

Lessons Learned

Identifying a representative set of training data requires a high initial time investment for the archivists. To support AVA’s learning process, the archives provided 3000 carefully verified appraisal decisions that were used to teach the tool how to differentiate between permanent and temporary content. In addition to the videos selected by AVA, it is important to ingest any available metadata about the full set of videos, such as dates, meeting titles, meeting room, participants, and others.

A human driven iterative training process is still required to continue training the tool and increase accuracy. Archivists need to plan for future developments and ensure that those plans reflect the current technology and are funded appropriately.

Our current use case is relatively simple, requiring evaluation of straightforward criteria. Scaling it to larger collections of mixed formats, etc. will likely present greater challenges, such as more AI bias because of human errors on the training data or because the training data is not representative enough.