« The cost of High Availability (HA) with Oracle | Main | Scalable federated security with Kerberos »

May 04, 2010

Business continuity with real-time data integration

Enterprises want to protect their data. As the appetite for data volumes grows, storage technology becomes a critical business asset on which business continuity relies. My recent survey in the medium-size enterprise segment shows the five dominant investment directions at the level of data management architecture: disaster recovery (DR), high availability (HA), backup, data processing performance and migration to more advanced databases.

This suggests that corporations generally have sufficiently structured data collections but are concerned with business continuity and continuous availability of data. What infrastructures can provide these assurances?

The most basic approach to provide data safety is hardware or software replication that automatically maintains a secondary copy of critical data. Home-made backup methods, based on open source software are also not unheard of.

Oracle customers have more options: Oracle Active Data Guard can be used if the source and copies are based on identical Oracle versions and data models, while Oracle RAC promises transparent application failover if the copies are in close proximity. Finally, as the Infrastructure-as-a-Service (IaaS) model wins the market for its price and elasticity, one can take advantage of a remote Cloud to host a secondary copy of the business-critical data.

In this post I want to focus on yet another option, and that is the Real-Time Data Integration model. As an example I am going to discuss Oracle GoldenGate, a solution that permits you to manage the data critical to your business in safety, ensuring business continuity without disruption even if the data is distributed among multiple, heterogeneous business applications and architectures.

Key facts about GoldenGate

By 2009, GoldenGate, a San Francisco-based vendor of real-time data integration software was still a modest size (by Silicon Valley standards), but already had a considerable market presence: 500 global customers including top commercial banks, telcos and retail companies.

What led Oracle to the acquisition? With its product portfolio, Oracle could already build advanced HA architectures for its customers, based on their own products.  However, Oracle could not directly compete compete in the domain of real time data replication.

How does it work? The secret is simple: GoldenGate can synchronize two heterogeneous databases, by reading the real-time history of all transactional changes at the source database. The delta is then  piped to the other database(s), again in real-time. In technical language, this is called Change Data Capture (CDC).

Simplicity is the feature of killer apps; GoldenGate guarantees transactional integrity when copying between source and target databases. With the simple deployment of  GoldenGate between databases, one can build a highly available architecture with each database holding accurate copies of each other's data.

Most importantly, GoldenGate permits active-active database replication. Imagine two separate branches of a corporation with separate IT systems (typical internal legacy of an acquisition process and a nightmare for sysadmins). With active-active, even though the two offices are not fully integrated, users can write to both locations at the same time, and the data will remain consistent.

Here we touch one of the core problems of distributed data storage systems, which is the issue of consistency between geographically separated data replicas. Oracle GoldenGate guarantees that even in an unstable environment where networks and host servers occasionally drop out, transactions will never be missed or skipped. Active-active is not without it’s challenges – collision detection must be developed, but Oracle GoldenGate provides a foundation upon which business logic can be easily built.

When to use real-time data integration

In the real-world, corporations use mixed IT architectures from multiple vendors (often a legacy of corporate history). When selecting a technology for a real-time data integration platform, the ease of integration across the spectrum of database and application platforms from multiple vendors is key. GoldenGate is no exception here; its Oracle license permits use with OracleDB and competitive products alike, knowing that this is the only way to maintain the competitive edge. Here are a few scenarios when real-time data integration can be considered:

  1. High-availability: Oracle GoldenGate will automatically maintain a live remote copy of your application’s interim data, so your business application can fail over  to the secondary storage during a disaster recovery scenario, with minimal down-time.
  2. Live migration: upgrade, migration or maintenance of a production system typically involves downtime. Oracle GoldenGate enables zero-downtime migration, as the new system can be populated with old system data without operational interruption of the former.
  3. Integration of heterogeneous systems: your applications rely on Oracle, MS SQL, Sybase, DB2 and what-not? A real-time data integration platform  can make them all operate on the same, shared data with minimal integration effort.
  4. Mergers, acquisitions and IT consolidation in a growing enterprise: before your final, uniform architecture will be devised, a CDC technique can quickly consolidate data from branches and departments. (By the way, we both know that there is no such thing as “final architecture”!)
  5. Query offloading: an interesting side-effect of sharing replicated data among multiple data marts is the improvement of OLTP performance and availability. Queries processed simultaneously by multiple servers execute faster and your reports are ready sooner.

When to use something else?

Savvy managers will carefully consider their options when building a real-time data integration architecture. For instance, care needs to be taken when planning for real-time data consolidation of multiple production systems, when at the same time thinking of it as a HA architecture. Can your business survive if one of them fails? This often depends on short and long-term investment management strategies.

Another issue addressed specifically to existing Oracle customers is how does GoldenGate coexist with other technologies, be it Oracle or non-Oracle that partially fulfill the same or similar functionality? You obviously don’t want to duplicate your investment. It is wise to learn and correctly differentiate between scenarios whereGoldenGate can complement RAC , Active Data Guard or other high-availability paradigms, and those where it is easier to use one of these products standalone. With Oracle licensing, these are not trivial questions and substantial cost savings can be made through informed decisions.

We will cover these subjects at the forthcoming seminar in Krakow, Poland, May 27th 2010.

Bookmark and Share


TrackBack URL for this entry:

Listed below are links to weblogs that reference Business continuity with real-time data integration:


Feed You can follow this conversation by subscribing to the comment feed for this post.

What is often overlooked is how conflict management is addressed: your application *MUST* deal with conflict itself! Do not rely on GG's conflict detection, you are swamped by the time it see it! GG doesn't "manage" conflict, it will only report it.
Stable, reliable replication is GoldenGate's bag, and it delivers there. I look forward to the day that GG integrates with Oracle RAC clusterware seamlessly (current implementation for clusters is quite rough).

Andy: very true, as I hinted "collision detection must be developed, but Oracle GoldenGate provides a foundation" for such app-specific business logic.

Active-active mode of operations is a theoretical and universal challenge, because of data synchronization. GoldenGate is no exception here.

Would you like to elaborate on how you'd like to see GG+RAC integrated? For simple HA scenarios, the two can almost be considered alternative, rather than complementary (my previous post discussed a highly available architecture based on RAC)

Though it was quite focused on one particular brand and product, the overall article was very informative and educational. I enjoyed it.


Do not await on GG's battle detection, you are ashore by the time it see it! GG doesn't "manage" conflict, it will alone address it.

Bottom Line: We need to educate the public that certain information can be passed harmlessly. Are there scoundrels out there abusing the system? You bet. But these decroted pieces of crap have been around for decades before the Internet. We should focus our resources on finding these slime bags and getting rid of them.

This was a very interesting read and I learned a lot about the subject of data safety. It makes sense to keep a second copy of enterprise data in case the primary copy is corrupted. Of course that would mean more costs for the business that has to invest in more storage hardware. However, as time goes by, the capacity of hardware is increasing all the time, making this step easier and cheaper for businesses. I am not familiar with the Oracle enterprise information system, so this analysis of the GoldenGate was very helpful for my studies. Thank you for the article and keep it up.

I enjoyed reading this post. But, it is important to include the Oracle NoSQL Database Community Edition 2.0 with improved Hadoop integration, elastic scaling and new APIs, including JSON and C support.

The comments to this entry are closed.