Gallery

Size Variation with LTFS Archiving

Recently we received an email from a customer who noticed a slight variation in the reporting of space used for an archive. Below is the question he asked and the answer we sent back. I thought it might help others when trying to fully understand LTFS archiving.

Our customer writes:

… I offloaded 2.20TB (2207.51GB) to the LTO6, the database details tells me it is actually 2.21TB in size, and the CSV report tells me it is 2.19TB used….

Imagine Products Response: 

Several things come into play when you’re talking about file sizes. Bottom line is that it’s not as simple as one might think, and file sizes are not a good indication of data matching.
First, please realize that a tape is NOT a hard disk. While LTFS is a great invention to allow us to “see” the contents represented similarly to what a hard disk might look like to the computer, a tape and a disk are physically different and how they actually store data is quite different.
Also please know that Finder uses Spotlight to index the contents of mounted volumes, and often yields very confusing if not nonsensical results when dealing with large data sets, especially if they’re being browsed or changed. And Spotlight doesn’t work well (or at all) with LTO tapes.
The short explanation is there’s a difference between “size” and “size on disk”.
How-To Geek
Why is There a Big Difference Between ‘Size’ and ‘Size on Disk’?
Most of the time, the values for ‘Size’ and ‘Size on Disk’ will be very close to matching when checking a folder or file’s size, but what if there is a huge discrepancy between the two?
Another thing that comes into play with tapes is the available and consumed space calculations depend upon responses from the deck. In the real world, tape sometimes has bad spots on it and the tape deck is designed to check for those and automatically skip bad sections. When that happens it simply rewrites the file it was working on to the next segment of tape and marks the bad section as deleted. This of course consumes what was thought to be usable space, but the drive doesn’t communicate that to our software so we really only have an approximation of how much data might fit on any given tape. To allow for this tolerance, we give PreRoll Post a cushion of 5% of the reported space–in other words, we won’t let you attempt to add more data to a tape than 95% of it’s reported available space. This reserve is purely to allot for any bad tape sections.
Anyway, more to the point, while hard disk allocated space is in 4KB chunks regardless of the actual data size, tape doesn’t behave in that manner. It is not exFAT formatted. So when you add lots of files (quantity more than overall aggregate size of them matters) this difference accumulates.
With an application like PreRoll Post, we use file copy routines to exactly copy the files and then double check them with checksums (that also test the byte sequences, not just total bytes). So, you can rest assured that the copies are 100% exactly all your data and an exact duplicate of them (regardless of the approximate size calculations).
About PreRoll Post: PreRoll Post is an LTFS archiving application optimized for the media and entertainment industry. PreRoll Post securely archives assets to LTO tapes or ODA cartridges using simple drag and drop functionality as well as checksum technology to ensure archives are 100% accurate. PreRoll Post uses the LTFS open-source so even tapes not created in PreRoll Post can be imported and retrieved (*only those using LTFS). PreRoll Post is compatible with any LTO tape drive as well as Sony’s Optical Disc Archive. For more information visit the Archive Home Page onwww.imagineproducts.com 
Gallery

Checksum Verification While Offloading

Watch an excerpt from the Moviola.com webinar: Data Wrangling in the Digital Age

Offloading camera cards properly can be one of the most important parts of the filmmaking process. Checksum verification helps ensure that those cards are copied 100% accurately. Learn more about why using finder is not secure, checksum values, different types of checksums and a few more helpful tips and tricks about the offloading process.

Verification

What do I mean by verify? Simply put, it’s determining that copied files exactly match originals.

Many people think checking total file sizes using Finder or Explorer assures they match. But the truth is unless there’s a catastrophic copy failure, Finder will report the allotted space for the copy not the actual file information. And, if you’re comparing contents of a folder or mounted volume, the file count and used space may be different due to hidden files.

And copying with just ‘Drag & Drop’ rarely gives any indication of copy failures and no assurances of byte sequences within the files. You can only know that by calculating the checksum values of the source and comparing it to the checksum of the copied file. While you could do that with Terminal commands or other labor-intensive means, from a practical standpoint you need a copy application that automates this function for you.

There are many different checksum types with various targeted uses. In the video industry however, the majority of workflows now rely on MD5 or the newer XXHash. Both are relatively quickly calculated and yield a high degree of certainty that two files exactly match. In other words, there’s a very low possibility with either of matching checksum values unless the files are in fact identical.

MD5 checksums typically are more CPU intensive than the XXHash, which relies more on system RAM for it’s processing. For this reason, computers with faster RAM hardware will see faster turn around on checksums with XXHash than using the MD5 type.

From a quality control point of view, sampling clips at beginning, middle and end helps give some assurance that the video files are not corrupt. But it’s also a good idea to spot check for things like unwanted items in view, etc.

Of course playing clips requires a player that supports the specific format. While each camera manufacturer generally offers a proprietary application for specific file types, there are a couple products like our HD-VU2™ software that’s designed to sample footage without transcoding or altering files.

Thanks for ready! #OffloadConfidently

 

Gallery

Manual Application Activation – for Firewall or Internet Problems

Screen Shot 2016-07-20 at 10.30.43 AM

Sometimes customers run into firewall or internet issues when trying to activate applications. Have no fear! There is a simple manual activation process that I am going to walk you through step by step, right now.

  • In the activation window select Manually Activate (for users with Firewall/Internet problems)
  • The manual activation window will then pop up with three spots for different information; serial number, ID number and personal activation number.

*SIDE NOTE: The ID Number is for that particular computer, this is a great way to keep track of your serial numbers. Create an excel spreadsheet and with the serial number, ID number and application for reference in the future. This is especially important for those that own multiple licenses of the same application.

  • Anyway, back to manual activation! From your phone or a computer that has internet access log into your account on our website. On the My Product page select Manually Activate button for the application you would like to activate.MA-01
  •  On the next page you will see your serial number (product code) at the top and then a place for your ID number (pictured above). In the Manual Activation window in the app, put your serial number in the top spot where it says …. serial number!
  • On the webpage put the ID number in from the application, then select the Activate Now button.
  • The next page will provide you with a personal activation number, put this number in the manual activation window in the personal activation number spot.
  • Select Activate and you are all set!Screen Shot 2016-07-20 at 11.09.39 AM

We hope you’ve enjoyed another fascinating (riveting even) and incredibly informative blog from Imagine Products!

Gallery

Staying Organized While Offloading

Watch an excerpt from the Moviola.com webinar: Data Wrangling in the Digital Age  

Offloading camera cards quickly and securely is possibly the most crucial part of the filmmaking process. Staying organized can make or break the offloading process. Don’t be the person who accidentally formats the wrong card and loses a days worth of media! Read on for more helpful organizational tips while offloading.

Tip 1

First, consider using low adhesive color-coded tape to help sort camera cards.

Have a SIMPLE procedure such as: Wrap cards with green tape if they’re empty and camera ready. Wrap with red if there’s a problem or “don’t use”.

Unwrapped cards would mean they’ve come out of the camera and need to be copied. Also flip the Lock switch on the card if it has one to indicate it contains data. In this scenario, never reformat an un-wrapped card without double-checking its contents and whether you have processed it.

Keep notes of where things were saved, what was done, anything unusual that’s happened, etc. It’s important to know where cards were offloaded. Track card serial numbers with content information. (Then if you discover a bad card you’ll know which clips double check.) An offload application that outputs copy reports is very useful for tracking purposes.

Tip 2

Have IN and OUT areas for cards to be copied and finished cards. Make sure everyone knows which is which—even label the areas. You could be plastic containers, anti-static bags, wherever—just be sure it’s clear.

Also give verbal confirmation: When you receive a card, confirm with the person it’s ready to be copied. When you give one, be specific and professional. Say “Okay to FORMAT”. Don’t say something ambiguous like “This is okay.” Etc.

When you receive a card, switch the copy protection to READ ONLY. Not only is this another visual signal that the card is ready to be copied, but on a practical note it prevents accidental erasure or additional files being added to the card.

If you’re using a Macintosh computer, locking the card also prevents Spotlight from adding it’s hidden index file to the card should you browse the contents in Finder. This may not sound like a big deal, but if the card is completely full this tiny little file can clip the end a big video file or even corrupted it. For this reason, it’s not a bad idea to disable Spotlight while offloading cards.

If you would like to learn more about data wrangling check out, Data Wrangling in the Digital Age Webinar on moviola.com http://tinyurl.com/z9w2598

You can also watch another excerpt from this webinar: Possible Offloading Bottlenecks and How to Eliminate Them 

Gallery

Possible Offloading Bottlenecks and How to Eliminate Them

Watch an excerpt from the Moviola.com webinar: Data Wrangling in the Digital Age  

Offloading camera cards quickly and securely is possibly the most crucial part of the filmmaking process. Recognizing and eliminating issues that may slow down this process can save valuable time, money and resources. In this blog we explore the bottlenecks of offloading and give several suggestions to offload as quick as possible.

Card Capability.

First, know the media type you’re working with and its expected performance.

You can expect a faster read than write of digital media. For example, a SanDisk CFast memory card writes at about 240MB/s but can read back more than double that speed at 515MB/s. Once you anticipate the media’s top performance, you can size the other components of your system.

Reader Speeds and Transfer Rates.

How are you going to connect the reader? For example, a FireWire800 can transfer about 100 MB/s while USB-3 can zip along at more than six times that at 625 MB/s. However, be aware that most portable computers have only one USB bus so each device plugged into a USB port splits the bandwidth.

Also realize that multiple slot readers are only as fast as the cable that connects them to your computer. So queuing up two SxS Pro cards that can offload at 150 MB/s in a reader means you need a computer connection and output hard disks that can handle over 300 MB/s to match the card’s speed.

Output Hard Disk Speeds and how they’re connected these days are the most common bottleneck. With a typical 7,200 RPM external disk you can expect about 100-120 MB/s write speeds, while an SSD drive can accept well over 400MB/s.

In general, computer resources such as CPU speed and RAM are generally not the limiting factors. What’s more important are the connections to the computer and the operating system’s memory management. Always use the latest operating system if possible—after all, there’s a reason Apple and Microsoft keep updating those!

If you would like to learn more about data wrangling check out the FREE webinar, Data Wrangling in the Digital Age on moviola.com. http://tinyurl.com/z9w2598