The intent of this page is to provide a forum to address various points of misinformation concerning PLOG processing, and thereby provide the reader with a better understanding of the issues involved with ADABAS Protection Log (PLOG) handling.
The following topics are presented and discussed:
If you have additional questions or issues that you would like to see addressed within this forum, please feel free to contact us.
To summarize, the challenges of PLOG data are not really any different than the basic issues presented in general data processing - and these issues CAN BE completely handled by in house programmers. Alternatively, a Software AG customer can utilize tools such as NatQuery and NatCDC to automatically deliver the required processing.
No Other Product Can Handle Changes In ADABAS
In addition to the NatWorks’ product NatCDC, there are at least two other products on the market today that can address proper PLOG handling, not to mention the fact that customers can even write these processes themselves through the use of the ADABAS utilities ADASEL, ADACDC or ADAPLP (all of which are provided free with ADABAS) and Natural. The simple fact of the matter is that there are alternatives available, and in proof of this, we offer the following links to two separate companies (besides NatWorks) which offer PLOG handling solutions:
At NatWorks, we specifically invite you to check out these alternatives, and compare and contrast the combination of NatQuery and NatCDC to them. After this comparison, we are confident that you will see the power, flexibility and cost-effectiveness of the NatQuery / NatCDC solution. For an idea of the performance characteristics of the NatCDC solution, please visit the PLOG Processing Information page.
Using ADASEL is a “severe handicap” and is an “inefficient use of CPU”
There are indeed challenges to overcome when handling PLOG data, and these topics are discussed in the above section entitled The Challenges in Handling PLOG Data
Of the five major challenges involved in handling PLOG data, the decompression challenge is the most significant. It is the decompression algorithms that cause the most CPU overhead when handling PLOG data - and there is ABSOLUTELY NO PRODUCT on the market today that will decompress PLOG data faster or more efficiently than ADASEL.
It must be remembered that PLOG processing is a constantly recurring activity on a system, and excessive CPU usage will have an impact on overall operation.
Rather than being a “severe handicap” or “inefficient use of CPU”, it is the intelligent use of PLOG utilities that offers incomparable performance and value over any third-party product that attempts to handle decompression on their own. By utilizing the ADASEL, ADACDC or ADAPLP utilities as an integral part of PLOG processing, the NatWorks tool, NatCDC, leverages what you already own and automatically builds processes that are routinely used.
Backed-Out Transactions Are A Problem
A “Backed-Out” transaction is a transaction that has been physically applied to the ADABAS database, but for whatever reason this transaction is subsequently reversed (removed) from ADABAS. There are several reasons why this might occur, however in most applications, this situation will be a relatively rare occurrence.
To understand why Backed-Out transactions occurring within an ADABAS PLOG is a concern but not necessarily a problem, one must first understand that ADABAS uses an “optimistic update” approach to transaction handling. This “optimistic update” approach means that when a Store, Update or Delete transaction occurs against ADABAS, ADABAS immediately applies these actions to the database - even though the physical command to make these transactions “stick” (I.E. an End Transaction command or “ET”) has not yet occurred. In this interim period, ADABAS will show this new data to all users if it is requested, even though it has NOT been “ETed”.
Prior to an End Transaction being issued, a Backout Transaction can occur either programatically by issuing a Backout Transaction or “BT” command (this would be a rather bad programming practice however), or this can automatically occur by ADABAS itself when it senses that too much time has elapsed since the Store, Update or Delete was issued with no “ET” or “BT” having been subsequently issued (this is referred to as a “Transaction Timeout”).
If a Backout occurs in ADABAS, whether programatically or automatically by ADABAS itself, the Backout will be reflected in the PLOG by the creation of Record Images that precisely reverse the previous transaction(s) that previously occurred. So: As long as whatever product / process that is being used to handle PLOG transactions properly processes the contents of a PLOG, these backouts will be properly handled and interpreted.
Transactions That Span PLOGS Are A Problem
As ADABAS operates (and assuming that Disk Logging is used as opposed to logging to Tape), ADABAS typically writes transactions that occur against ADABAS to one of two dedicated disk files that contain PLOG dataset. When one PLOG disk file is completely filled, ADABAS automatically switches to writing transactional data to the second PLOG dataset, and at the same time initiates a process that copies the contents of the first PLOG dataset to tape or disc. In this manner, ADABAS should always be able to write transactional changes, and previously written transactional changes can be appropriately archived.
The “problem” referred to above has two aspects:
In regards to the first aspect of this “problem”, the answer lies in understanding how ADABAS’ “optimistic update” works. This then is the same issue as described in the Backed-Out Transactions section, and if your tools have the capacity, is not really a problem.
In regards to the second aspect of this “problem”, per Software AG log# 124049 and in reference to ADABAS version 6.2.2, an Update transaction will NEVER be split across two PLOGS. Subsequent to providing NatWorks with this information, Software AG subsequently reversed themselves on this information and then indicated that a BI from an Update transaction COULD be written to one PLOG dataset while the Update’s corresponding AI could be written to the next PLOG dataset.
Given that Software AG now indicates that a split could occur, the ability to properly handle these split transactions can be accomplished in one of two ways:
Transaction Referential Integrity Is A problem
This issue is in essence the same “problem” as described in Backed-Out Transactions Are A Problem.
The crux of this issue is that ADABAS uses an “optimistic approach” (transactions are made available systemwide immediately once they are made and prior to them being ET’ed or BT’ed) to transaction handling, and virtually all relational databases use a “pessimistic approach” (meaning that transactions are not made available systemwide until after the transaction is fully finalized).
Regardless of approach, there will always be situations in which the referential integrity of specific records in a target warehouse are compromised at the point that a PLOG switch occurs, and it is currently unworkable (for NatWorks or any other PLOG vendor) to attempt to guarantee “referential integrity” on a relational database that uses a “pessimistic approach” to transaction handling when data is coming from ADABAS’ “optimistic approach”.
In light of this, we at NatWorks feel that an ADABAS Change Data Capture solution for ADABAS should embrace ADABAS’ “optimistic approach”, an approach that provides data that exactly reflects the status of an ADABAS database at the particular moment that a PLOG switch occurs. This approach then results in a “True” point-in-time of ADABAS.
What if I Only Want “Committed” Transactions in my Warehouse?
If it is a requirement that the target Data Warehouse / Data Mart should only be populated with ADABAS transactions that were committed (versus populating the target Data Warehouse / Data Mart with “optimistic” transactions that may not have been committed): Software AG provides the necessary tool to do this in the ADABAS utility, ADACDC.
The ADACDC utility processes PLOGs in a similar fashion to ADASEL (identifying transactions of interest and then decompressing these transactions), however ADACDC will “hold” transactions that have not yet been ET’ed whereas ADASEL makes no attempt to hold them. ADACDC then processes those transactions that were “held” against the next PLOG dataset to be processed, and then matches these “held” transactions to their respective commits.
Another difference between ADACDC and ADASEL is that ADACDC will automatically produce a “Delta” of transactions for you - whereas when utilizing ADASEL this “Delta” must be programatically handled (optional NatCDC process).
If processing only “committed” transactions into your warehouse is a requirement, then NatCDC can handle the output of ADACDC as easily as the output of ADASEL. NatCDC will provide the required processing to convert variable-length records to fixed length format, drop any unneeded fields, and convert ADABAS data types into ASCII equivalents, however there is no requirement for a Sort step or to deduce a “Delta”, when using the ADACDC utility.
United We Stand