Vendor Neutral Archiving

By George Florentine, VP of Technology

The idea of vendor neutral archiving often comes up once an organization has begun archiving data. They realize that getting locked in to one vendor doesn’t support a long-term archiving strategy. Let’s look at what prompts the need for vendor neutral archiving, its history, and steps you can take to avoid vendor lock-in.

Why Vendor Neutral Archiving Is a Thing

As we’ve discussed in previous blogs, many organizations start their archiving projects due to a specific business event – adoption of a new enterprise system such as an accounting, human resource, or ERP application, adoption of an industry-specific system such as an electronic health record system, or an M&A event requiring consolidation of systems.

After archiving data from an individual system, the organization will start to realize that they have many systems that have the following archiving requirements:

• Long-term data management
• Business continuity
• Cost recovery
• Enabling faster innovation
• Risk reduction
• Compliance management

Companies expand their thinking from “Let’s archive this one system” to “Let’s set up an archiving program and feed the program with candidate legacy systems as driven by business events.” Many of these systems come out of legacy environments with arcane infrastructure needs – decades old database technologies, outdated messaging protocols and operating systems, unsupported application run time environments, etc.

Somewhere in the group, someone has a great (?) idea – “Let’s build/use a vendor neutral archive!”

You can understand the motivations:

• Future-proof the archive by making it independent from a vendor’s product roadmap
• Improve the long-term financial stability of the archiving system by avoiding a dependency on a vendor’s changing support pricing model
• Simplify deployment model cost and complexity by eliminating the need for vendor private run time components.

Vendor Neutral Archiving: A Look Back

In the mid-2000s there was quite a bit of work both in industry and standards bodies to articulate standards and architecture to support a vendor neutral archiving vision. But this work effort gradually died out.

Why? The notion of a vendor neutral archive had some strong barriers to adoption:

• Financial disincentive. What vendor wants to contribute to building a product that takes away their product differentiation?
• New vendor offerings bring new capabilities into the market – speed, scale, cost of ownership, usability, security, etc. What customer wants to deploy a system with a predicted lifetime of 10-20 years knowing that new vendor innovations cannot be leveraged?

Our Approach: Build a Better Mousetrap

As we watched (and contributed) to this shift in thinking, we realized that we could make a better mousetrap – one that enables ongoing innovation, leverages industry standards, future-proofs data, and protects customers from vendor lock in and the associated concerns around support pricing.

We call this archiving approach and architecture Optimized Archiving. We developed these program-level principles to guide our thinking:

• XML Data Interchange. XML is an industry standard for defining document structures and allows for the use of a single character encoding system and the modeling of arbitrarily complex record structures. By enforcing an XML in<->out capability in our solutions we guarantee that data remains vendor neutral.

• Standard image and document definition. XML is good for storing representations of structured document and data records but not good for storing page-oriented documents and images. We use industry standard formats such as PDF, PNG, and JPEG that are supported by a wide range of vendor and open-source toolkits.

• Pluggable storage systems. Archiving systems tend to be big with sophisticated query and data update requirements (remember retention policies!). To meet these requirements, a solution must use database systems, but again, we don’t want to get locked into a single vendor. Defining a pluggable mechanism to leverage existing storage technologies (MarkLogic, OpenText™ InfoArchive, MongoDB, Cloud SQL) enables us to get the performance and features we need while avoiding a long-term vendor dependency.

• Micro services architecture for component features. We recognize that security, naming, messaging, queuing, service discovery, etc., will all evolve over time. We want a solution that allows these features to be pluggable and discoverable at run-time. Remember our two guiding principles – leverage best in class features while enabling future improvements.
• Container support. We believe that the notion of container-based application deployment represents a long-term shift in how applications are deployed and managed. Docker has won the container wars and will be a dominant run time deployment technology for the next 10-15 years. We use that today but if another technology gains sufficient market share, we can readjust our build and deployment tools to leverage the new technologies.

With this approach we can leverage best-in-class technologies while maintaining our long-term vendor independence. We incent and reward vendors that develop great technologies and deliver these benefits to our customers while avoiding long-term vendor entanglements.

But how hard would it be to build such a system? Two years ago, we started work on this challenge and are now shipping Flatirons Digital Hub. An architectural diagram of our archiving solution is shown in Figure 1.