The following steps provide an overview at a more technical level of how the Testimony product operates.
Testimony is installed as a third-party add-on solution to a central system (typically your SAP Solution Manager system). Once installed, you are able to remotely install a “listener agent” into your production system(s). This listener agent has no impact upon your production system and lies dormant once installed.
Test Plan Configuration
All processes that occur within Testimony are done under the guise of a “Test Plan”. This represents the regression test cycle(s) that are currently being performed for the necessary scenario (e.g. HANA DB migration, AWS re-platforming, technical upgrade etc). Within the central Testimony system, you must configure a test plan and provide some basic information relating to it (e.g. what the system(s) involved for recording and playback, which users are involved from a testing perspective). RFC destinations must be setup to allow Testimony to trigger both the recording and playback processes.
A setup activity is then run to prepare the system to be recorded. After this point, the system is ready to enable you to activate the “recording” function in Testimony. Please note that until you perform this activation, there is zero impact on production service.
When you are ready to perform a regression test cycle, the operator is able to activate the recording process. This will record all technical operations upon the SAP system that it is installed into including activities such as SAP GUI dialog transactions and RFC / BAPI calls. Soon after activating the recording (within the next 2 hours), the entire SAP production system is copied to an image by the basis administration team. It is typically envisaged that a period of approximately 24 hours of operational activity would be recorded. At the end of the 24 hour recording period, the operator deactivates the recording process. Due to the unique manner in which Testimony operates, the impact on the production system is near zero (in terms of response times and database performance).
The recording process has in effect automatically generated an entire test script library covering over N% of the customers actual SAP system usage (where N would be expected to range from 70% to 90%). After the recording process is finished, the testing team can understand the level of coverage they have with the recording based upon the usage information of what the customer actually uses over the past M months (where M is typically 3 months). If key transactions are missing (e.g. month end processes) then the coverage analysis within Testimony can provide this information and additional scripts can be run to cover those processes that are missing.
Once the recording and coverage process are run, then it is possible to execute the playback process. The production system(s) that were originally copied in the recording phase must be started. The Testimony playback agent must be setup on the target system and “drone” agents run upon supporting infrastructure including Windows based machines. The drone agents simulate the actions of users as well as other external systems during the playback. Changes that are being made must be deployed to the target system (e.g. upgrade process, project release transports deployed, database migrated). Once the landscape is ready, the playback process can begin. This is simply a process of starting the “execution queue” and allowing it to execute (technically). If issues arise, notifications can be sent to appropriate test team members. The queue can be restarted if problems are detected.
After the play-back process has completed, the test team can review the results. They are specifically looking for issues that have arisen during the playback where the “expected output” differs from the “actual output”. Any discrepancies identified can be automatically generated as defects. The testing and development team can review these defects (once created) and provide a resolution. Once the bulk of defects are resolved, a new test cycle can be created and the system copy image restored. The playback process can be repeated to ensure that defects identified have been correctly resolved.