Challenges of digital media preservation

1. Challenges of Digital Media Preservation

Karen Cariani, Director
Media Library and Archives
Dave MacCarn, Chief Technologist

2. Who we are: WGBH Media Library and Archives

2

3. Transition challenges (Analog to Digital)

Preservation needs are more
complicated
— New and changing content formats
— Network connections
— Software
— Storage media
— Hardware
Access expectations challenging
— Faster access
— Anywhere, anytime
3

4. Content formats

4
H D Acq u i sit ion Cod e cs
V id e o Sa m p le
Au d io
1920
1080p
1920
1280
960
1280
M bps
1080i
1440
720p
Sa m p le
√
√
√
√
3:1:1
4:2:2
4:2:2
4:2:2
4:2:2
4:2:2
4:2:2
4:2:2
√
RGB or 4:2:2
√
4:2:0
4:2:0
4:2:2
4:2:0
4:2:0
4:2:2
4:2:2
2
4
8
2
2
4
2
MP1
16
24
16
16
16
16
4:2:2
4
16
4:2:2*
4:2:0
4:2:0
4:2:0
4:2:0**
2
2
2
2
2
AC3
AC3
AC3
16
16
1 0 Bit
#
Bit s
√
√
√
4
8
8
8
4
4
8
8
20
16
16
16
16
16
24
24
12
4
24
Co n t a in e r
D CT
[1]
HDCam
DVCProHD@24p
DVCProHD-720p
DVCProHD-1080i
Avid DNxHD
Avid DNxHD
Apple ProRes
Apple ProRes HQ
140
40
100
100
145
220
145
220
√
√
√
√
√
√
√
√
√
√
√
√
W a v e le t
Red
224-336
√
M PEG2 Lo n g GOP
HDV
XDCamHD
XDCam422
XDCamEX
XDCamEX
GFCam
Canon C300
25
18-35
50
25
35
50
50
√
√
√
√
√
√
√
100
√
√
√
√
√
M PEG2 I - f r a m e
GFCam
H .2 6 4
AVCHD PS
AVCHD
AVCHD@24
Canon 5DMKII
Nikon D800
28
24
24
38
24
H.2 6 4 I - fram e
AVCIntra
AVCIntra
50
100
√
√
√
√
√
√
√
√
√
√
√
√
√
440
√
√
880
√
√
√
Tape
DV-AVI, DV-DIF, MXF, QuickTime & Tape
DV-AVI, DV-DIF, MXF, QuickTime & Tape
DV-AVI, DV-DIF, MXF, QuickTime & Tape
MXF & QuickTime
MXF & QuickTime
QuickTime
QuickTime
REDCODE & QuickTime
M2T, MXF, QuickTime & Tape
DV-AVI, MP4, MXF & QuickTime
DV-AVI, MP4, MXF & QuickTime
DV-AVI, MP4, MXF & QuickTime
DV-AVI, MP4, MXF & QuickTime
MXF
MXF
MXF
MTS, MP4
MTS, MP4 & QuickTime
MTS, MP4 & QuickTime
QuickTime
QuickTime
4:2:0
4:2:2
√
√
2
2
16
16
MXF
4:2:2/4:4:4
√
12
24
DPX, Tape
4:2:2/4:4:4
√
12
24
DPX, Tape
MXF
M PEG4 St u d io Pr o f ile
[2]
HDCamSR
[2]
HDCamSR-HQ
√
*Sony FS100 HDMI output
** 4:2:2 HDMI output
[1] Tape format for comparison
[2] Tape with DPX file out
D. MacCarn, WGBH

5. Storage and retrieval

How do we:
Capture the audio and video generated by myriad cameras
Store the project information to allow potential re-edit
Store files with rich, meaningful metadata
Store born-digital materials
Display and retrieve born-digital materials
5

6. Access: Organizational Issues

Metadata
Descriptive metadata
— Need description for video to be useful, findable
— How to capture that
— How to make sure it is linked to video files
6

7. Folder Structure

7
Create folders by card
— Assign unique number
— Continue numbers
— Add description
— Place ENTIRE card
contents into this
folder!!

8. Original footage

© 2011 WGBH
8

9. Proposed tapeless workflow

Create a mapping document between filemaker and
DAM
Used to generate an xml stylesheet
Video is ingested simultaneously with the metadata
from filemaker using the xml stylesheet
Technical metadata is ingested simultaneously with the
video and production data using the xml generated by
the source digital files
9

10. Challenges - again

Access issues
— File size
— Formats – to playback
— Useable — Search/findable
Metadata
Organize files
Preservation issues
— Copies
— Formats – for migration
— Being able to play again later
— Speed of access (big file size) – to use/process
— Migration ease
10

11. Software /Network

File management
— Where are the files?
Needed for access to files
— Large preservation files
— Smaller access, proxy files
Network speed
— Larger files, need faster
network to meet speed
expectations
11

12. Issues with current file mgmt systems/software

Preservation not a priority
Interface issues
— Access vs. Preservation
IT relationship
— Tech support
— Vendor reliance issues
— Need library based system for Archivist needs rather than traditional
IT company needs
Expense
— License cost
— Development
— Customizations
12

13. Access

13
Can find
Can view
Can select
Can get out of
system
Can reuse in
editing system

14. Preservation Needs

Multiple Copies
Validity
Bit quality checks
Long lasting storage
Regular migration
Persistence
14

15. Challenges of preservation and access

For preservation
— Want to capture as close to original as possible
— Originals may be many different formats
— Will need to make sure you can export and use different formats in
future
— File format issues
— Fixity check big files
For access
— Want one consistent format for playback/access
— Needs to be easy to migrate, use
15

16. What makes video different?

Preservation files are large
— Uncompressed
— Slow to move around
Need proxy files for viewing
— Smaller size for quick transport
over network
Complicated formats
— Not just one file type
— Codecs, wrappers, frame speed,
etc
16

17.

Technology Mix:
17

18. Hydra project

Combine preservation system with access system
Better interface
Flexible design
Easy to evolve
18

19. Insert graphic

Blacklight Hydra heads
Hydra mgmt layer
Fedora repository
HSM storage system
19

20. Fundamental Assumption #1

No single system can provide the full range of
repository-based solutions for a given institution’s needs,
…yet sustainable solutions require a
common repository infrastructure.
20

21. Fundamental Assumption #2

No single institution can resource the
development of a full range of solutions on its
own,
—…yet each needs the flexibility to tailor
solutions to local demands and workflows.
21

22. Hydra Philosophy -- Community

• An open architecture, with many contributors to a
common core
• Collaboratively built “solution bundles” that can be
adapted and modified to suit local needs
• A community of developers and adopters extending
and enhancing the core
• “If you want to go fast, go alone. If you want to go far,
go together.”
22

23.

CRUD in Repositories
Create/Submit/Edit
(CUD)
Search/View
(R)
Repository/
Persistent Storage

24.

Major Hydra Components
hydra-head
Rails Plugin
(CUD)
Blacklight
(Read
(R)
Only)
Solrizer
Fedora
Solr

25. Hardware/Storage media: HSM

Access
— Online
XX bytes Spinning disk
— Offline
— Nearline
Preservation (offline)
— Robotic tape library system
— LT04 data tapes
— 2 copies
— One stored off site
Migration needs 3-5 years
— Both tape migration to newer formats
— Technology migration

26. New Storage Types and Costs

Need hierarchical storage (HSM)
— Video files are large
— Spinning disks are expensive
— Tape can help save cost
— Tape copies/migration can be automated
26

27. New Storage Types and Costs

But HSM has licensing issues
— Some systems cost by gigabyte managed
— Need Open source alternative
27

28. Q & A

Q&A
Karen: [email protected]
Dave: [email protected]
28