Gallery

Apple File System: What do you need to know?

The new Apple File System (APFS) has been in beta for a while now and Apple enthusiasts everywhere expect it very soon. But what exactly can we expect and what’s the point?

The point is that the APFS will run on WatchOS, iOS, tvOS and macOS making the whole Apple universe more scalable and a more seamless experience all the way around.

But why the change? Well, the current system, HFS+, is over 30 years old. I can tell you from personal experience that things over 30 years old tend to not work as well as they used to! It was designed when we were still using floppy and hard drives and a rigid data structure but times, they are a’changing and we need backwards and forwards compatibility and APFS is more flexible and dynamic.

Now don’t freak out, if you have apps that run on HFS+ (and we all do), APFS will not kill them. APFS is designed to both support and replace HFS+, you will likely see no change at all.

Before I go further I want to make sure to give credit where credit is due, this clever, well written blog is all mine but all the information within came straight from the horse’s mouth, more accurately, from Eric Tamura and Dominic Giampaolo’s mouths.

If you are interested in watching the 36 minutes video that goes all through the new APFS you can do so here. They do some cool demos in this and it really shows how much faster APFS is versus HFS+.

apple-file-system-100705705-large.jpg

What these two smartie pants shared is as followed, I am not going to go over every new feature but I will touch on most of them:

  • Space Sharing: Do you have more than one partition? Cool, you’ll love this. Currently, free space on one partition doesn’t translate to more space on another partition.. but with APFS it will! Now there will be ‘containers’ which will hold all the available space allowing any partition to grow at any time.
  • Flash/SSD optimization: most of the products have solid state drive, APFS is designed with that in mind. It’s predecessor was created when floppy disks were prevalent – can you even find those anymore?? HFS+ has basically been pushed and pulled as far as it will go and it’s just not feasible or financially responsible to continue with it.
  • Crash protection: no one ever plans on crashing but it happens, hell we buy insurance for just such an occasion. APFS takes crash protection to a whole new level. The new metadata scheme safeguards that the writes to storage are always in sync with writes to the file system journal even in the event of a power failure.
  • Low-latency design: this basically means that apps will open faster (yay for app developers – ahem) and data will be delivered faster (again, yay!).
  • Strong encryption: APFS will support metadata encryption, per-file encryption and per-extent encryption (each “region” of a file can be encrypted). APFS is the only file system currently allowing users this level of encryption – hence why we love Apple so much – they give us what we want!
  • 64-bit inode numbers: Now I’ll be honest, I had no earthly idea what this meant! Fortunately our developers do. Inodes are data structures that store metadata about objects (we love metadata) as opposed to 32-bit, meaning it is capable of storing 9 quintillion files…. that’s a thousand raised to the power of six (1018)!

So now you are in the know and we can patiently wait for the new Apple File System to drop together.

I had to sneak this throwback Apple in just for fun! article_img#apple #APFScountdown

Michelle's Headshot  written by Michelle Maddox,

Marketing Director for Imagine Products

 

Gallery

Checksums Part 2: Define and Decide

This is a multi part blog. As more blogs are posted, links to those posts will be included in this blog. Before reading further, we encourage you to begin with Checksums Part 1: The 5 W’s which defines what a checksum in and why it’s important in the media and entertainment industry.

For Part 2, we asked YOU what you were most interested in and many answered they would like each checksum defined and explained as to which is better to use in different situations. Obviously you are all using ShotPut Pro for your offloading right? Perhaps you are researching which offloading application is right for you. This blog will also discuss appropriately comparing offload applications and their use of checksums.

We will not reinvent the wheel but we will give credit where credit it due! All sources used to help explain and define checksums have been appropriately included. If you’re looking for some light reading… These are not be for you! But if you are looking for more in-depth information, they have plenty.

xxHash – xxHash is an extremely fast, non-cryptographic hash algorithm, working at speeds close to RAM limits. It is proposed in two flavors, 32 and 64 bits. (SMHasher on github.io)

For ShotPut Pro, ShotSum and PreRoll Post we use xxHash 64 bit. We recommend using XXHash as the checksum type unless you have a requirement for some other type. XXHash can out perform MD5 for example because it can go at the speed of your RAM whereas MD5 is a CPU dependent process.

MD5 – The MD5 algorithm is a widely used hash function producing a 128-bit hash value. It is optimal as a checksum to verify data integrity. (Wikipedia MD5)

For years MD5 was the fastest and most secure checksum available. Although xxHash is becoming more widely used there are still many companies that require the MD5 checksum for data integrity.

checksum-menu

The primary things to consider when choosing a checksum is how fast is it and when talking about files, what is the chance of a checksum collision?  The collision chance is the probability that two different files will map to the same checksum value.  xxHash is great because it is fast while still having a low probability of collision.  Many even consider hash functions more secure than a byte by byte comparison because the chance that hardware gives back the wrong results is in many cases higher than the chance of an checksum collision.

SHA-1 (Secure Hash Algorithm 1) – is a cryptographic hash function. SHA-1 produces a 160-bit (20byte) hash values known as a message digest. A SHA-1 hash value is typically rendered as a hexadecimal number, 40 digits long. (Wikipedia SHA-1)

SHA-2 (Secure Hash Algorithm 2) 256 and 512 – this is an upgrade to SHA-1 and includes six hash functions, Imagine Products applications offer two of the six. Cryptographic hash functions are mathematical operations run on digital data; by comparing the computed “hash” (the output from execution of the algorithm) to a known and expected hash value, a person can determine the data’s integrity. (Wikipedia SHA-2)

MD5 and some of the SHA checksum algorithms are sometimes still used as a requirement for different insurance companies or by the government because they are older and established, but they were designed to be cryptographic hashes.  Cryptographic hashes are those originally designed to store things like passwords.  They were meant to be complex and sometimes even slow to ensure that passwords were safe.  This isn’t ideal for many of us which is why xxHash was created.

Anymore, unless you have a specific requirement from a vendor, we recommend xxHash or sometimes MD5 because the speed of the others is usually not worth the trade off in collision space. In the cases of SSD to SSD copies you most likely will see a performance hit with any of our algorithms other than xxHash.

 

Comparing Checksum Applications

It’s a good idea to choose the right tools for the job. Obviously we think our products are the best (and so do thousands of others!) but it’s a good idea to test different workflow applications to be sure you are getting what you need to ensure data integrity and accuracy.

Here’s one thing to remember…

Computers are designed with a combination of caches along the data-handling stream. They’re present in hard disks, in the connection ports and inside the computer’s operating system. The idea is to speed up the return of data requests for ‘known’ recently accessed items.

Think of how your web browser caches web pages you’ve previously visited. Caching browser history allows them to quickly present recent pages when requested again without having to go back and download the entire page each time.

The Apple operating system has a similar methodology (as do hard drives themselves). Items recently accessed are kept in a revolving cache of RAM for fast presentation.

So when an application asks the operating system to read back the most recent file from the output hard disk the Mac OS says “Oh! No need to go get that again, I have a copy right here!” and simply returns the cached information. This is great if you’re not trying to actually compare and verify one copy to another. Instead, it’s just a repeat of the source file and not a fresh full read of the file from the output disk, which is meaningless for verification purposes and doesn’t even offer the security of comparing file sizes.

In fact, with Apple operating systems you can not obtain a hard disk read of files (instead of from cache) without explicitly circumventing the cache–a method only seasoned programmers would be aware of or those incredibly interested in how checksums actually work – like yourselves!activity-monitor-icon

To test different offloading applications, open the Activity Monitor utility on the Mac. Boot it up and click on ‘Disk’. Then open the Terminal utility and flush the cache by typing in the command “SUDO PURGE”.

Once you’re ready, do a reasonable size offload–say 15 GB. Then look at the Disk Read and Write GBs in the lower right table. In order to perform checksum comparisons the Read GBs should be double the Write GBs. That’s because you’re reading once from the source, writing once to the output drive, then reading back from the output drive to calculate the checksums.

If the application or method you’re considering using in your workflow doesn’t have double the read GBs compared to the write GBs then it’s not actually retrieving the disk’s content to compare with the source files and is less secure than true checksum comparisons. In other words, the final destination of your files may not actually match the source – this is a big problem if you are truly concerned with data integrity. Remember, if it seems too good to be true – it probably is.

That’s all for Checksums Part 2. If you have questions or feedback leave them here, email us or post it on any of our social media pages. Part 3 is coming soon – let us know what you want to learn more about. And always – Offload Confidently!

Gallery

Imagine Products’ Connection to the Olympics

The Olympics are always a fun and exciting time for the world. It’s amazing to come together as one human race and celebrate the incredible achievements of talented athletes. In 2002, Imagine Products was asked to travel to the Salt Lake City games to assist with the logging of that years Olympic footage using Image Mine®, our video cataloging software. Dan Montgomery, President and Owner of Imagine Products, recounts his time at the 2002 Salt Lake City Olympics.

Slcolympics

As I sit watching the Rio summer games, and truthfully whenever any Olympics rolls around, I can’t help but reminisce about my experiences with the Salt Lake Winter Olympics in February of 2002. 

Our sales manager “made the sale” the summer prior to the Olympics. It was the largest single sale to date and of course being the Olympics made it even more exciting. The agreement was that we would customize a version of Image Mine for them and provide on site support during the games. This last part was like a cherry on top as far as we were concerned! Who hasn’t wanted to go to the Olympics.

Then 9/11 happened and the whole world changed.

Instantly the thought of traveling, which was once a no brainer especially in the business world, became a serious question of safety. For our small company it meant the loss of our sales manager whose young wife was afraid of him traveling, and understandably so, it was a scary time for most of the world. Of course as the President and the next best

FullSizeRender
My badge to get me “behind the scenes” – or really just to the broadcasting center.

 

 person knowledgeable to fulfill the contract, I went. 

I arrived about a week prior to the opening ceremonies and was housed at a Ramada Inn about 10 blocks from the Salt Palace Convention Center that had been transformed into the broadcasting center. On the first bus ride to the center I noticed lots of concrete road barricades, chain link fences, barbed wire, etc. At first I thought, “Wow, it’s just like Indiana. They tear up the roads just before the Indy 500!” But of course it was all about security. 

 

baracades

Barricades outside the Salt Palace Center. Photo Attributed to www.sbe24.org

There were three rings of barricades around the convention center, bomb-sniffing dogs, airport style metal detectors and police with assault rifles. Oh! And did I mention snipers on rooftops around the city? Yeah, it was like that.

Inside the center was a grid of temporary “offices” built from drywall. It was like a large city with security guards positioned at each intersection of the grid and they had to have eyes on each other at all times.

There were two scary moments leading up to the games. One was someone setting off a door alarm going outside to smoke. The other was detected “explosives” which turned out to be the fireworks they were bringing in for the opening ceremonies. Both times a very loud alarm would sound, the security guards would take off running and the building went into immediate lockdown. Fortunately both instances were short lived and once the games actually started there were no more issues.

Video collection and distribution was organized through a shell company, International Sports Broadcasting, based in Salt Lake. While NBC also had their own production team on site, the ISB was tasked with shooting all events without regard or preference to the athletes and creating the International Olympic Committee’s visual archives of the games. When I say ‘without preference’ I mean each country represented is primarily interested in following their athletes regardless of ranking, so it’s a somewhat monumental task to provide that kind of coverage. 

backuproom
The Backup Room in the Broadcasting Center. Photo Attributed to T.Buckingham Thomas

To do this the ISB contracted videographers from all over the United States to assist in the various venues and broadcasting tasks. Additionally, they hired about 50 senior broadcast students from various universities to perform the cataloging tasks. It was a great gig for a college student – you got to see the games, pad your resume and earn $50 a day! 

The ISB had set up 28 video logging stations. Each had a Gateway laptop with a USB 1.0 video capture input and our logging software on them. Gateway was a sponsor of the Olympics that year and all the equipment was ‘payment in kind’. The USB throughput was slow considering the video was analog, we are talking 20 frames per second! But since the software was just grabbing thumbnails, it was adequate. This was the first Olympics that thumbnails were used for archiving, before that all archives included descriptive text but no pictures.

 

DVCPro
Panasonic DVCPro 50 Photo Attributed to www.abelcine.com

The workstations also had two Panasonic DVCPro tape decks plus two VHS tape decks. Yes, you read that right! Panasonic had its own support area with about 20 techs solving any issues that arose immediately. Three high quality copies were made, one came from the truck or venue and two VHS tapes, one with timecode and one without. I was really surprised and asked “Why VHS?” and they said because VHS was the lowest common denominator for universal viewing footage.  

Olympic Fun Fact: No two modern Olympics to that point had been shot on the same video format. As far as I know, this is still true!

Remember I said part of the agreement was customization? ISB had paid 3M for a custom full-sheet label that printed top and side labels for all 5 tapes, plus the box labels all at once. Our tracking software printed them with information extracted from the cataloging database. The Tracker module also was used to keep inventory of blank tapes as they left to the various venues. It supported barcodes to check the In & Out as the used tapes returned to the center.

Control room 1
I met a lot of really great people when I was in Salt Lake. The Olympics has a tendency to bring out the best in people.

 

I found it fascinating that the students had been practicing cataloging for days before I arrived. It was the first time I didn’t have to teach anyone how to use our software; instead they were showing me tips and tricks!

Since they would be cataloging video LIVE it was important for them to know the rules of the events they were logging. And they needed to know the key players, what to watch for and what drama might ensue. To prepare for this the organizers had been pumping video to the center from the prior Olympic games as if it were happening live and the students were cataloging them as if they were as well. Pretty smart way to prepare, considering the day the games actually started – it was like any other and they did what they’d been doing all along. 

The students were versed in several sports each so they could credibly catalog them. Each morning during the games there was a big white board with the assignments for the day. It was hilarious to see the ones assigned to curling – big sighs and grumbles, “Curling again! Ugh.”

My own kids at that time were teenagers and when their school teachers found out I was at the Olympics of course the questions flew. My daughter after having explained my involvement several times decided it was easier to tell them I was an Olympic class curler! After all, what else would a 50 year old, overweight diabetic be able to compete in?!?

My actual job at the Olympics? Remember the “support” portion of the contract, turned out to be much like the Maytag repairman. Nothing failed and no support was needed so for almost a month it was me sleeping at my desk or watching the Olympics – on a tv screen – exactly like I do from home 😂

Control room 2
Can you see Muhammad Ali holding the torch? This was the practice footage used from the 1996 Olympic games to get everyone ready for live logging. From the beginning I had plenty of time to take pictures and chat with my fellow loggers because we’ve always made quality software!

Good luck to all the Olympic competitors!

We hope you have enjoyed this little walk down Olympic-Memory lane.