Contact us
, , , ,

SAP CPI – Archive Message Processing Logs (MPLs) to SAP BTP Document Management Service (DMS)

Bhavesh Kantilal

By.

Bhavesh Kantilal

•

min read

Share

Post Views: 2,053

I had previously posted SAP CPI – Message Monitoring – Standard Out of the Box Features to use in your Iflows. Standard Monitor on CPI are quite powerful and along with SAP CPI – Reusable Groovy Script to Log Payload as Attachments; we have a comprehensible solution for monitoring in CPI. But, that’s not without limitations.

Introduction

SAP Standard Monitoring features work fine but they have some limitations,

  • SAP stores the Messages Processing Log Archives ( MPLs) for 30 days.
  • When you use Attachments to log your messages to the DB; you have a 30 GB limit on the DB & 1 GB per day and thereafter you can get a circuit breaker on your Message Processing Logs.

SAP provides you with a convenient option to ship your Message Processing Logs ( along with any attachments ) to a External Document Management Service. In this post we will explore how to enable this option and what this option brings to the table.

Pre-Requisites

It is assumed that you have enabled SAP BTP Document Management Service Integration Option / Repository Option as per this post of mine SAP BTP DMS – Integration & Repository Option- Initial Set Up

Note the following details from the Service Key created for BTP DMS

  • ecmservice -> url -> Will be referred to as ecm_url  herewith
  • uaa -> clientid
  • uaa -> clientsecret
  • uaa -> url -> Will be referred to as uaa_url  herewith
  • RepositoryId -> Refer to post in Pre-req to get the RepositoryId

Create Destination – CloudIntegration_LogArchive

Create a Destination CloudIntegration_LogArchive in SAP BTP in your CPI / Integration Suite Sub Account.

  • Name: CloudIntegration_LogArchive
  • Type: HTTP
  • URL: ecm_url/browser
  • ProxyType: Internet
  • Authentication: OAUTH2ClientCredentials
  • Client ID : client_id
  • Client Secret: client_secret
  • Token Service URL: uaa_url
  • Token Service User: client_id
  • Token Service Password: client_secret
  • Additional Properties
    • RepositoryId : <<RepositoryId created in DMS>>

Create Service Instance and Service Key

Login to your SAP BTP Cockpit , go to your SAP CPI / Integration Suite subaccount.

Create ServiceInstance

Create a Service Instance with below details.

  • Service: Process Integration Runtime
  • Plan : api
  • InstanceName: <<anyName>> CPI_RT_API_Archival in my case
  • Roles
    • AuthGroup_Administrator
    • DataArchivingActivate
    • DataArchivingRead

Create Service Key

  • Service Key Name : CPI_RT_API_Archival_SK ( Can be any name)
  • Key Type: ClientId/Secret

Parameters to Note in Service Key

  • clientid
  • clientsecret
  • url
  • tokenurl

Activate Archival Settings in Postman

Get Token

  • URL : {{tokenurl}}?grant_type=client_credentials
  • Authentication : Basic
    • User : clientid
    • Password: clientsecret
  • Get access_token from response

Activate Archiving

  • URL : {{url}}/api/v1/activateArchivingConfiguration
  • Authentication : Bearer Token
    • Use access_token from Previous call
  • Method : Post

Your Archival is now Active

Check Archival Settings in CPI

Logout and Log back into your CPI URL. Navigate to Monitoring -> Manage Integration Content. You should now see Options for Archive Data in your each Iflow.

As you would notice Archival Settings are an Iflow Level. You can choose to Activate and Deactivate Archiving at an Individual Iflow.

You can choose to Activate Archival of

  • Sender Channel Messages
  • Receiver Channel Messages
  • Log Attachments
  • Persist Messages ( Messages of Persist Step Type)

You can choose to Archive the data you want to Archive

And Thats it , your job is done! Messages will now be Archived automatically.

How Does Archival Work?

Once you activate the Archival of Messages in your Iflow in CPI Monitor, SAP automatically schedules a Job to Archive the messages. The default Archival Threshold is 7 days, i.e., Messages are pushed to the Archive after 7 Days. There is no UI to schedule this job from a CPI Operations. A Ticket needs to be raised to SAP to change the Archival Period from 7 days to any duration that you want. The ticket has to be sent to SAP CPI Operations.

How are messages stored in the Archive

You can use a CMIS Client like Apache Chemistry CMIS Workbench as shown in my post : SAP BTP DMS – Integration & Repository Option- Initial Set Up.

When you connect to the CMIS Repository, you will see folders now getting created in below hierarchy.

  • YYYY
    • MM
      • DD

For example in my CMIS Repository you can see below folders : YYYY – 2023, MM – 06 ( For June), DD- 14,19 for Messages Archived from 19/06/2023 and 14/06/2023

Within each folder you will see a .zip file for each Message that is Archived. The name of the file is the MessageId.zip

Note the Type of the CMIS Message – mpl:message.

Each Message has Properties that can then be queried. See the custom properties highlighted below.

What Does the Archival File Contain?

This depends on your configuration for each Iflow but for our case where we selected SenderChannelMessages, ReceiverChannelMessages,Attachments

  • Custom Headers are Archived
  • The Attachments are in the folder “StepId” ( CallActivity_5 in my case)
  • Sender and Receiver Channel Messages are in the folder “MessageFlow_” (MessageFlow_4 in my case)
  • Headers are logged ( Non sensitive ones ) for Sender channel and receiver channel messages.

How Do I query or search for messages in the Archive

Using APIs

You can use Postman and the DMS APIs for it. See a example below where I use the following Query to select all messages / files for Iflow: ArchivingTest

select * from mpl:message where mpl:iFlowSymbolicName like 'ArchivingTest'
  • URL : https://{{ecm_url}}/browser/{{repository_id}}?cmisselector=query&q=select * from mpl:message where mpl:iFlowSymbolicName like ‘ArchivingTest’
  • Operation: GET
  • Authentication: Bearer Token

Using CMIS Workbench

What happens to messages in CPI Message Monitor after they are Archived

  • A CPI Message that is Archived to BTP DMS is not automatically deleted from the CPI Database.
  • CPI continues to persist the message for the default duration of 30 days in the Database.
  • The CPI message monitoring UI only shows the messages from the Database.
  • After 30 days a message that is Deleted from the Database is not shown in the UI even though it is in the Archive.
  • In other words, CPI message monitoring UI only uses its down Database to display the messages. For the messages in DMS, you need to build your own UI.
  • Once a message is Archived by CPI, the message status will change from “Archive Pending” to “Archived“

What happens to when you have multiple Receiver Channels in CPI – Eg: Request Reply

In the case of a Iflow that has a Request Reply step in the flow and we have selected the option to Archive SenderChannel Messages and ReceiverChannel Messages in the Archive Data for the iflow , CPI will also Archive the Input and Output Payload for each RequestReply Step in the iflow.

Below is the naming convention of the folder

  • CallActivity_ -> Attachments
  • MessageFlow_ -> Send, Receive, RequestReply and Sender Channel Messages

See example below.

How do I check the Archival Status?

SAP provides some platform APIs to check the Archival Status and the Archival Job Status

Check Archiving Configuration

  • URL : {{url}}/api/v1/ArchivingConfigurations(‘tenant-name’) –> leave tenant-name as is
  • Authentication : Bearer Token (Use access_token from Previous call)
  • Method: GET

Check Archival KPIs / Job Status

  • URL : {{url}}/api/v1/ArchivingKeyPerformanceIndicators
  • Authentication : Bearer Token (Use access_token from Previous call)
  • Method: GET

The API returns values like

  • RunStart, RunDurationInMinutes etc
  • MplsToBeArchived, MplsArchived, MplsArchivedUntilDate
  • These are then used in the next run to get the delta and so on.

Final Thoughts

  • CPI provides a very effective out of the box Archiving Solution to BTP DMS ( or any CMIS Compliant DMS).
  • Messages are archived and settings can be changed on the fly at a Individual Iflow level.
  • Limitation ( at the time of writing this post in July 2023 ) exists where the Messages Archived are not shown in the standard UI ,i.e., there is no out of the box UI to connect and search for messages from the Archive.
  • The lack of a out of the box UI for CPI message monitor to connect to the External Archive and search makes its real world usage a challenge.
  • Hopefully there exists in the future releases an option to select in the CPI UI, selection to select from MPL Archives to BTP DMS rather than own data base so customers do not need to build their own UIs for this.
, , , , ,

Bhavesh Kantilal