Gallery

Checksums Part 1: The 5 W’s

This a multi part blog. As more blogs are posted, links to those posts will be included in this blog.

 

checksums-01

What is a checksum?

A checksum is a string of numbers and letters that uniquely identify a particular file regardless of the type, ie .mov, .ari, .dng, .png, .txt. Think of them like a fingerprint; no two fingers have the same fingerprint (nope, not even identical twins!) and no two random files have the same checksum; different files = different checksums. This allows people (obviously with the help of computer applications) to compare checksum values to ensure a 100% exact copy of a file is made when duplicating, or offloading files.

Why are checksums important in the media and entertainment industry?

Video files are made up of bytes. Many times when files, especially video files, are being moved around the bytes can be corrupted, inverted or lost completely. This can cause lags or glitches in video during playback. Simple applications like Finder or Explorer do not use checksums. Instead, they “look” at the files or folders being moved or copied and say, “ok, looks like 10 GBs. Yeah I’ve got room here for that amount” and then they dump the files. The problem occurs when only 9.8 GBs are moved and 0.2 GBs are lost OR the total 10 GBs are moved but bytes 3.2 and 3.3 are switched around so that 3.3 comes before 3.2…. I think you can see the problem here! Finder and Explorer have no safeguards put in place to ensure this doesn’t happen.

When and where are checksum appropriate to use?

Checksums can be used when transferring any file type. Many real estate, doctors and lawyers offices usesnape checksums when moving files from one place to another. You can understand why losing files or folders for these industries would be disastrous. In the media and entertainment world, losing files equals losing shots. Sometimes you can not replicate those shots or they would be too costly to replicate. Either way, the short answer to the question is ALWAYS. Any time files or folders are being moved, checksums should be used. Sorry (not sorry) for the Harry Potter reference, I couldn’t resist!

Who should use checksums?

Honestly, if you are serious about data integrity – you should be using checksums. Not only that but make sure you have a report with all the checksum values to cover your own butt. Many studios and insurance companies are requiring offload applications like ShotPut Pro that use checksum algorithms. If there is not a report you can’t prove that checksums were used.

That’s all for this post! I hope it’s cleared up what checksums are and why they are important to use. We will be delving more deeply into the world of checksums in the near future. If you have a specific question you would like answered please post a comment here, tweet or Facebook us!

Gallery

Size Variation with LTFS Archiving

Recently we received an email from a customer who noticed a slight variation in the reporting of space used for an archive. Below is the question he asked and the answer we sent back. I thought it might help others when trying to fully understand LTFS archiving.

Our customer writes:

… I offloaded 2.20TB (2207.51GB) to the LTO6, the database details tells me it is actually 2.21TB in size, and the CSV report tells me it is 2.19TB used….

Imagine Products Response: 

Several things come into play when you’re talking about file sizes. Bottom line is that it’s not as simple as one might think, and file sizes are not a good indication of data matching.
First, please realize that a tape is NOT a hard disk. While LTFS is a great invention to allow us to “see” the contents represented similarly to what a hard disk might look like to the computer, a tape and a disk are physically different and how they actually store data is quite different.
Also please know that Finder uses Spotlight to index the contents of mounted volumes, and often yields very confusing if not nonsensical results when dealing with large data sets, especially if they’re being browsed or changed. And Spotlight doesn’t work well (or at all) with LTO tapes.
The short explanation is there’s a difference between “size” and “size on disk”.
How-To Geek
Why is There a Big Difference Between ‘Size’ and ‘Size on Disk’?
Most of the time, the values for ‘Size’ and ‘Size on Disk’ will be very close to matching when checking a folder or file’s size, but what if there is a huge discrepancy between the two?
Another thing that comes into play with tapes is the available and consumed space calculations depend upon responses from the deck. In the real world, tape sometimes has bad spots on it and the tape deck is designed to check for those and automatically skip bad sections. When that happens it simply rewrites the file it was working on to the next segment of tape and marks the bad section as deleted. This of course consumes what was thought to be usable space, but the drive doesn’t communicate that to our software so we really only have an approximation of how much data might fit on any given tape. To allow for this tolerance, we give PreRoll Post a cushion of 5% of the reported space–in other words, we won’t let you attempt to add more data to a tape than 95% of it’s reported available space. This reserve is purely to allot for any bad tape sections.
Anyway, more to the point, while hard disk allocated space is in 4KB chunks regardless of the actual data size, tape doesn’t behave in that manner. It is not exFAT formatted. So when you add lots of files (quantity more than overall aggregate size of them matters) this difference accumulates.
With an application like PreRoll Post, we use file copy routines to exactly copy the files and then double check them with checksums (that also test the byte sequences, not just total bytes). So, you can rest assured that the copies are 100% exactly all your data and an exact duplicate of them (regardless of the approximate size calculations).
About PreRoll Post: PreRoll Post is an LTFS archiving application optimized for the media and entertainment industry. PreRoll Post securely archives assets to LTO tapes or ODA cartridges using simple drag and drop functionality as well as checksum technology to ensure archives are 100% accurate. PreRoll Post uses the LTFS open-source so even tapes not created in PreRoll Post can be imported and retrieved (*only those using LTFS). PreRoll Post is compatible with any LTO tape drive as well as Sony’s Optical Disc Archive. For more information visit the Archive Home Page onwww.imagineproducts.com 
Gallery

Manual Application Activation – for Firewall or Internet Problems

Screen Shot 2016-07-20 at 10.30.43 AM

Sometimes customers run into firewall or internet issues when trying to activate applications. Have no fear! There is a simple manual activation process that I am going to walk you through step by step, right now.

  • In the activation window select Manually Activate (for users with Firewall/Internet problems)
  • The manual activation window will then pop up with three spots for different information; serial number, ID number and personal activation number.

*SIDE NOTE: The ID Number is for that particular computer, this is a great way to keep track of your serial numbers. Create an excel spreadsheet and with the serial number, ID number and application for reference in the future. This is especially important for those that own multiple licenses of the same application.

  • Anyway, back to manual activation! From your phone or a computer that has internet access log into your account on our website. On the My Product page select Manually Activate button for the application you would like to activate.MA-01
  •  On the next page you will see your serial number (product code) at the top and then a place for your ID number (pictured above). In the Manual Activation window in the app, put your serial number in the top spot where it says …. serial number!
  • On the webpage put the ID number in from the application, then select the Activate Now button.
  • The next page will provide you with a personal activation number, put this number in the manual activation window in the personal activation number spot.
  • Select Activate and you are all set!Screen Shot 2016-07-20 at 11.09.39 AM

We hope you’ve enjoyed another fascinating (riveting even) and incredibly informative blog from Imagine Products!

Gallery

Staying Organized While Offloading

Watch an excerpt from the Moviola.com webinar: Data Wrangling in the Digital Age  

Offloading camera cards quickly and securely is possibly the most crucial part of the filmmaking process. Staying organized can make or break the offloading process. Don’t be the person who accidentally formats the wrong card and loses a days worth of media! Read on for more helpful organizational tips while offloading.

Tip 1

First, consider using low adhesive color-coded tape to help sort camera cards.

Have a SIMPLE procedure such as: Wrap cards with green tape if they’re empty and camera ready. Wrap with red if there’s a problem or “don’t use”.

Unwrapped cards would mean they’ve come out of the camera and need to be copied. Also flip the Lock switch on the card if it has one to indicate it contains data. In this scenario, never reformat an un-wrapped card without double-checking its contents and whether you have processed it.

Keep notes of where things were saved, what was done, anything unusual that’s happened, etc. It’s important to know where cards were offloaded. Track card serial numbers with content information. (Then if you discover a bad card you’ll know which clips double check.) An offload application that outputs copy reports is very useful for tracking purposes.

Tip 2

Have IN and OUT areas for cards to be copied and finished cards. Make sure everyone knows which is which—even label the areas. You could be plastic containers, anti-static bags, wherever—just be sure it’s clear.

Also give verbal confirmation: When you receive a card, confirm with the person it’s ready to be copied. When you give one, be specific and professional. Say “Okay to FORMAT”. Don’t say something ambiguous like “This is okay.” Etc.

When you receive a card, switch the copy protection to READ ONLY. Not only is this another visual signal that the card is ready to be copied, but on a practical note it prevents accidental erasure or additional files being added to the card.

If you’re using a Macintosh computer, locking the card also prevents Spotlight from adding it’s hidden index file to the card should you browse the contents in Finder. This may not sound like a big deal, but if the card is completely full this tiny little file can clip the end a big video file or even corrupted it. For this reason, it’s not a bad idea to disable Spotlight while offloading cards.

If you would like to learn more about data wrangling check out, Data Wrangling in the Digital Age Webinar on moviola.com http://tinyurl.com/z9w2598

You can also watch another excerpt from this webinar: Possible Offloading Bottlenecks and How to Eliminate Them 

Gallery

Possible Offloading Bottlenecks and How to Eliminate Them

Watch an excerpt from the Moviola.com webinar: Data Wrangling in the Digital Age  

Offloading camera cards quickly and securely is possibly the most crucial part of the filmmaking process. Recognizing and eliminating issues that may slow down this process can save valuable time, money and resources. In this blog we explore the bottlenecks of offloading and give several suggestions to offload as quick as possible.

Card Capability.

First, know the media type you’re working with and its expected performance.

You can expect a faster read than write of digital media. For example, a SanDisk CFast memory card writes at about 240MB/s but can read back more than double that speed at 515MB/s. Once you anticipate the media’s top performance, you can size the other components of your system.

Reader Speeds and Transfer Rates.

How are you going to connect the reader? For example, a FireWire800 can transfer about 100 MB/s while USB-3 can zip along at more than six times that at 625 MB/s. However, be aware that most portable computers have only one USB bus so each device plugged into a USB port splits the bandwidth.

Also realize that multiple slot readers are only as fast as the cable that connects them to your computer. So queuing up two SxS Pro cards that can offload at 150 MB/s in a reader means you need a computer connection and output hard disks that can handle over 300 MB/s to match the card’s speed.

Output Hard Disk Speeds and how they’re connected these days are the most common bottleneck. With a typical 7,200 RPM external disk you can expect about 100-120 MB/s write speeds, while an SSD drive can accept well over 400MB/s.

In general, computer resources such as CPU speed and RAM are generally not the limiting factors. What’s more important are the connections to the computer and the operating system’s memory management. Always use the latest operating system if possible—after all, there’s a reason Apple and Microsoft keep updating those!

If you would like to learn more about data wrangling check out the FREE webinar, Data Wrangling in the Digital Age on moviola.com. http://tinyurl.com/z9w2598

Gallery

LTO Best Practices Part 2: Advanced LTFS Software Can Be the Key to Success

Before you read this, Read Part One first!

The free software that comes with an LTO tape drive is meant only to get you up and running with LTFS so that you can see the contents of your tape in Finder or Explorer. Free software lets you browse the content, but it doesn’t make the content searchable, it doesn’t guide you through the process, and retrieval is excruciatingly slow.

On the other hand, having software that organizes files are keeps track of their positions on the tape is important for performance. Such software can quickly locate files and put them in a logical sequence for optimum retrieval and restore speeds.

Beyond the performance benefits, paid software can walk you through the whole process in order to optimize the archive for video and avoid the pitfalls mentioned above. The idea is to create a good backup from the beginning to prevent problems down the line.

For instance, some paid software forces uncompressed mode during formatting and also automatically formats the tape to meet Discovery Channel’s file-structure specifications, which seem to be the most stringent in the industry right now.

Paid software also makes it easier when it’s time to copy files onto tape. Remember the rule about illegal characters? Advanced LTFS applications analyze names and flag problems with illegal characters so you can fix them before you start the backup. With certain software, you can even configure it to replace illegal characters automatically during the backup process and then restore those characters when you restore the files. This capability is helpful in cases where it’s important to keep the same file name regardless of illegal characters, such as when linked files are referenced by other media.

Likewise, if the files you’re copying contain references or links, the software checks to see if they are valid. If not, it will flag the problem so that you can address it before starting the backup.

It is helpful to have LTFS software that is video-aware. That is, it always knows the typical video formats, the different types of data coming out of different cameras, and how to break up or keep together different data sets when spanning multiple tapes. Video-aware LTFS software can also extract thumbnails and even play proxies before ever pulling anything off the tape.

Finally, unlike free LTFS software, paid software compares source and copied files and performs checksums on the data, a common and often required practice in the postproduction industry. Checksums provide assurance that the source and copied files match exactly and didn’t get corrupted during the backup process. PreRoll Post takes checksums a step further by calculating a checksum again after the file is restored. This extra checksum calculation helps ensure the file wasn’t corrupted during restoration.

It pays to be organized and judicious about what to back up. Most people back up far more than they can use. For example, it almost never makes sense to back up hidden files because they will be incorrect when restored.

Remember the rule of two: It’s not a true archive without at least two copies of the content – most likely on two or more sets of LTO tapes – stored in two different buildings.

LTO and LTFS create a long-term, reliable, economical storage solution. If you’ve never backed up to LTO before, there is a slight learning curve, but with some common sense, best practices, and the right tools, you can easily set up your archive to deliver worthwhile results down the road … with far more reliability than you would ever get from hard disk.

#archivesecurely

Gallery

LTO Best Practices Part 1: You’ve Settled on LTO/LTFS. Now What?

LTO7_giant

Once you’ve decided to create an LTO/LTFS archive, there are some things you need to consider in order to be successful with it. It’s not the same as copying files onto hard disk and sticking it in a storage closet. The mechanics are different, so there are a few more steps, and there are rules associated with LTFS that require some forethought.

One thing to know is that LTO tapes are raw media that do not come formatted, so you have to format them before you can use them. You can format using terminal commands – a cumbersome process that few would undertake – or, when formatting for LTFS, you can do it using LTFS software. After that, any hardware that has LTFS software installed will be able to read the LTO tape. One of the defining characteristics of LTFS is that in the process of formatting, it partitions the tape so that one section stores the actual content and the other section stores an index of metadata. The index allows the tape to be self-describing, which improves archive management. It also helps ensure organizations are compliant with broadcast networks that require indexed tapes.

Another consideration is whether to format the drive in compressed mode, and the short answer is “no.” Operating in compressed mode is useful in the IT industry when backing up things like documents and text files. But in the media industry, where nearly all video files are already compressed into a format that saves space, there’s little to be gained by formatting a tape in compressed mode. In fact, it can slow down the writing and retrieval processes considerably.

Besides formatting considerations, there are rules related to LTFS that affect how you handle your archives. LTFS has its roots in the IT world, where most servers are running UNIX operating systems. In UNIX – and therefore in LTFS – certain characters, such as ampersands and backslashes, are reserved for formatting purposes and are off-limits in folder and file names. For those of us in the video world, who are used to using just about any character we want, those restrictions can cause trouble when setting up an LTFS file structure. That’s why it’s important to know those “illegal” characters and be diligent when naming files. If you use illegal characters, then the archive retrieval process could crash later.

A related issue, and a common stumbling point for most people, is trying to back up files that contain links or references to media that are not part of the backup. Such files can cause restoration to fail because the application is trying to find a file that doesn’t exist. The same thing can happen with files that contain absolute paths, which trace back to a specific point on a hard disk but lead nowhere on a tape.

For more LTO tips and tricks read part two of LTO Best Practices. #archivesecurely

Gallery

FAQ: Fixes for Slow Verification Speed with PreRoll Post Windows

PRP new SMALL-01
Are you archiving with PreRoll Post Windows and feel like your verification speeds are too slow? Here are some helpful tips on how you might be able to increase verification time.
Slow verification speed is often caused by another software attempting to access the LTO drive while PreRollPost is doing its verification.  The most likely cause of this is anti-malware software scanning the LTO deck and treating it like a random access hard disk and not a linear tape drive.
To fix this please exclude your tape drive and PreRollPost.exe from the anti malware software.  To do this with the built in windows defender please follow these steps:
1.    In search type in “Windows Defender”
2.    Open the windows defender application
3.    Click settings in the top right hand corner
4.    Click “Add an exclusion” under the Exclusions category
5.    Click exclude a process and type in PreRollPost.exe
This fix should be immediate and can be done while a verification is in progress.
Similar steps may need to be followed if you have additional anti-malware software installed. Archive Securely!
About PreRoll Post: PreRoll Post is an LTFS archiving application optimized for the media and entertainment industry. PreRoll Post securely archives assets to LTO tapes or ODA cartridges using simple drag and drop functionality as well as checksum technology to ensure archives are 100% accurate. PreRoll Post uses the LTFS open-source so even tapes not created in PreRoll Post can be imported and retrieved (*only those using LTFS). PreRoll Post is compatible with any LTO tape drive as well as Sony’s Optical Disc Archive. For more information visit the Archive Home Page on www.imagineproducts.com 
Gallery

ShotPut Pro Increasing Speed

SPP5 new LARGE MW-01When offloading, speed and accuracy is paramount. If you are already using ShotPut Pro (with checksums) then you’ve got the accuracy portion completely covered. *If you aren’t using checksums, shame on you! What’s the point of offloading without checksums? You have no way of knowing if your camera cards (or original source) has been copied correctly or completely! Watch for our upcoming blog on checksums to learn more.

OK, back to assuming you use checksums with ShotPut Pro to offload your media. The next most important part would be speed. We are always “running out of time” or “looking to save time” or worse “totally out of time, I need it now!” Here are some helpful tips on how to be sure ShotPut Pro is running at top speed. With proper setup ShotPut shouldn’t take more than 20% longer than a drag and drop in Finder. The increase is related to the checksums which I think we already covered was crucial and we’re assuming you are all using them, right… RIGHT?!?

  1. Always be sure you are using the latest version. Under the application menu select ‘Check for Updates’ to see if you are using the most up to date version. We often make changes and many of those are to keep up with OS changes.
  2. Purge the Log History frequently. When you’re copying a large number of files the backend SQLite database file can get bulky and slow things down. *Are you receiving the error “ShotPutPro5.sqlite” couldn’t be opened – refer to this blog post for simple to follow instructions.
  3. Double check that your buffering size is appropriate for your equipment, you can find this under the application menu > Preferences > Advanced. DO NOT set the buffering higher than medium unless you are using SSD drives or have very fast connections to everything (like eSATA, SAS, TB, etc.). Setting the buffer too high for your equipment can cause extremely slow copy times because the software can overwhelm the ports on your computer.
  4. Be sure you are using the most up to date OS, particularly on Macintosh computers. Older OS versions are slower and some are even buggy causing even more slowness.
  5. USB connections: Remember that USB ports often share the same bus and having multiple devices plugged in splits
  6. FireWire Daisy Chains: Same as USB, each daisy chained item splits the bandwidth and quickly deteriorates your throughput.
  7. Card Readers: If you’re copying CF cards, please know the readers can overheat. Have more than one available to rotate between copies to allow for cooling time.
  8. Multiple Slot Readers: While there are several multi-slot card readers on the market, these are for convenience more that speed. Most reader and card drives do not handle reading two cards at once efficiently. It can really slow down the process.
  9. Multiple Copies: Yes, ShotPut can make more than one copy from the same source at a time and it does so efficiently. However, if you’re trying to copy more than one volume at a time to the same spinning media output drives they can ‘compete’ with each other to control the drive.
  10. DO NOT browse files with Finder while ShotPut is copying. Doing so adds more read/writes to the disks and adds Spotlight overhead.
  11. Speaking of which, turn off Spotlight if possible (with an app like Spotless).
  12. Internal Drives: if possible do not copy to an internal disk. There typically run slower at 5400 RPM and are being accessed for the copy processes too so copying to it will slow you down.
  13. ALWAYS set system power preferences to never sleep while copying. Also use a UPS for both the computer, card readers if any, and external drives.
  14. Hidden Files: Don’t copy hidden files unless absolutely necessary. There are never needed for card media and will only slow you down, and may cause errors if you browse the card while copying.

Offload Confidently!

About ShotPut Pro: ShotPut Pro is an offloading application optimized for the media and entertainment industry. ShotPut Pro quickly and securely offloads any file, folder or media using drag and drop functionality and checksum technology. Offload from multiple locations, to multiple locations automatically with watch folders. ShotPut Pro will send email or text notifications when the offload is complete and generate exportable offload logs and reports. For more information visit the Offload Home Page on www.imagineproducts.com 

Gallery

PreRoll Post: Adding/Removing LTO Tapes or Tape Content

PRP new LARGE all-01

One of the greatest features of PreRoll Post is it’s database. Here you will find every LTO tape ever created in PreRoll Post regardless if the tape is mounted or not. This is incredibly useful when it comes time to retrieve content. But for one reason or another you might need to delete a tape from your database OR add tape that wasn’t created in PreRoll Post to your database. This post will quickly walk you through that process.
To Delete:
Select the tape in the ‘Retrieve’ window and click the delete button on your keyboard, or right mouse click and choose Delete. This doesn’t delete anything from the tape, just your view of it in the Retrieve dialog. (This feature is handy for tapes you’re delivering to clients that you no longer wish to be able to search their contents.)
To Add:
Mount the tape and go to ‘Retrieve’ window. Right click on it and choose IMPORT FILES. That reads the index on the tape and uses it to populate the Retrieve database information.
See! I told you it was easy 🙂 This process can be used in both the Macintosh and Windows version. Thanks for reading, #archivesecurely
About PreRoll Post: PreRoll Post is an LTFS archiving application optimized for the media and entertainment industry. PreRoll Post securely archives assets to LTO tapes or ODA cartridges using simple drag and drop functionality as well as checksum technology to ensure archives are 100% accurate. PreRoll Post uses the LTFS open-source so even tapes not created in PreRoll Post can be imported and retrieved (*only those using LTFS). PreRoll Post is compatible with any LTO tape drive as well as Sony’s Optical Disc Archive. For more information visit the Archive Home Page on www.imagineproducts.com