Data Deduplication: The Humble Digital Hero We Needed

With such an abundance of critically important info saved on our various devices, it’s been a no-brainer to consistently back up that data. Remember the days of forgotten manual saving and cruelly timed power outages? It’s okay to shudder. 

Now we auto-back up everything from our emails and word docs to photos and work folders. Google Docs and Drive have changed the game for the better in this way, and collectively, an unthinkable amount of data gets backed up every day. Because it happens so easily in the background (from our perspective as users) you may not not realize how much data has been repeatedly copied and saved for your benefit.

Over time, your data storage becomes needlessly weighed down by superfluous copies. This can actually cost a business money – to cope with the excess, data requirements become larger and time is lost in the extra data processing. But as a business, you’ve got a server to run and revenue to pull… So what to do?

Enter Data Deduplication

Data deduplication is the process of deleting redundant copies and thereby minimizing processing time/effort for a given software system. When you back up your software system, what’s happening is that you are copying and storing these big old sets of data. Before too long, this tends to require a slow, inconveniently ponderous chunk of data storage. So data deduplication is how you optimize and streamline that storage, which happens by making sure that only one unique instance of any given datum is copied and subsequently stored.

If you’re pulling your data from a single source, data deduplication significantly improves efficiency. With boatloads of identical data in different places, your whole system will likely get bogged down.

How Data Deduplication Works

As a process, data deduplication is actually not too complicated. And when you back up your computer, this is a great time to employ a little deduplication action. Imagine an email server with like 50 different instances of the same 2 MB attached file. With no data duplication, this is the kind of thing that happens – that is, it’s what happens when everyone backs up their own inbox, and this is how you end up with 50 instances of the same stupid file across the whole server, which means you then need 100 MB of storage space for something that should literally only take up 2% of what it’s currently occupying. 

Data deduplication means ensuring that but one nice little instance of the file gets stored; subsequently backed up identical  instances are not stored, which reduces demand on your server’s bandwidth.

The Evolution of Data Deduplication

Not all deduplication techniques are identical. At first, data deduping did its early job of reducing required storage capacity, making data backup servers more reliable. Certain parties like Data Domain improved this model using target- and variable-block-based approaches that only only needed you to back up altered data segments—instead of all segments.

How do you back up an increasing load of data across a network without hurting said network’s performance? Variable-block and source-based deduplication compress data before it even departs the server, which minimizes traffic across the network—not to mention reducing the sheer amount of store data stored on a disk and the time taken to perform the backup. Deduplication has become more than merely saving and storing it now addresses overall performance from network to network, giving data the chance for a timely backup—even in limited-bandwidth environments. Data deduplication has even since evolved to eliminate redundancy at the level of the object—versus that of the file—and address the needs of users on a literally global scale.