Introduction
Recent enhancements of the Trading Partner management provides a comprehensive payload logging and monitoring solution for B2B Messages in CPI. B2B Monitoring in CPI as the name implies is heavily tailored for B2B transactions ( read EDI over AS2 ) but it provides an out of the Box UI and Payload Display. Below is how t typical B2B Monitoring in CPI looks,
As customers move forward in their PO to CPI Migration and build classic point to point Integrations, the need for a Payload logging solution for A2A flows continues to be a missing gap. Alternates exist as I have demonstrated in these posts of mine ( see links below ) ( and many other CPI posts for Payload Logging ) but they are “half baked” and have limitations. This post provides another approach to do Payload Logging but leveraging the B2B Monitoring Capabilities of CPI and out of the box API’s.
- SAP CPI – Reusable Groovy Script to Log Payload as Attachments
- SAP CPI – Reusable Groovy Script to Log Payload as Attachments – Turn off Logging at a Tenant Level
- SAP CPI – Message Monitoring – Standard Out of the Box Features to use in your Iflows
The Solution at a Gist
- SAP has recently released a updated package for B2B Trading Partner Management Capabilities – Cloud Integration – Trading Partner Management V2. This package contains Reusable groovy scripts that are used for B2B Monitoring and Payload Logging.
- This Solution leverages these Groovy Scripts as a input template and adapts them for A2A Monitoring.
- Standard Application Monitors of CPI described in this link act as the Dynamic Inputs to the Groovy Script SAP CPI – Message Monitoring – Standard Out of the Box Features to use in your Iflows namely
- SAP_Sender
- SAP_Receiver
- SAP_MessageType
- SAP_ApplicationID
- Integration Flow Name and Message CorrelationID are also available as Searchable Parameters.
- The Script provides options to
- Log Input Payload
- Log Output Payload
- Update Status of the Message
- Handle Retry Messages and their Logs
- Handle Exception Messages and their Logs.
The Solution in the B2B Monitoring
Successful Message
Failed Message
The Details – Pre-requistes
- Ensure that you have enabled Trading Partner Management capabilities in your Integration Suite Tenant.
The Details – The Groovy Script
The Groovy script uses the B2B Scripts from SAP as a example and are simplified to read the headers and properties and log the payload.
The Script Functions that I have defined are self explanatory
- LogIncomingPayload – Mandatory – Has to be the 1st Script to be called. This logs the Incoming Payload and forms the B2B Document Properties.
- LogAfterMappingPayload – Optional – Logs the after mapping Payload
- LogStatusCompleted – Mandatory for Success Scenarios – Logs the Status of the Message in the B2B Monitor as Completed.
- LogStatusError – Mandatory for Error Scenarios – Logs the Status of the Message in the B2B Monitor as Error and logs the Error Message.
- LogStatusRetry –Mandatory for Retry Scenarios – Logs the Status of the Message in the B2B Monitor as Retry for JMS and Data Store based retries.
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import groovy.json.*;
import com.sap.it.api.pd.PartnerDirectoryService;
import com.sap.it.api.ITApiFactory;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import org.osgi.framework.FrameworkUtil;
import com.sap.gateway.ip.core.customdev.util.Message;
import com.sap.it.op.b2b.monitor.api.*;
import com.sap.it.op.b2b.monitor.api.events.*;
import java.nio.charset.StandardCharsets;
import java.time.LocalDateTime
import java.time.format.DateTimeFormatter
import com.sap.it.api.pd.BinaryData;
import groovy.json.JsonSlurper;
def Message LogIncomingPayload(Message message) {
def headers = message.getHeaders();
def bundleContext = FrameworkUtil.getBundle(Class.forName("com.sap.gateway.ip.core.customdev.util.Message")).getBundleContext();
def serviceRef = bundleContext.getServiceReference(Class.forName("com.sap.it.op.b2b.monitor.api.B2BMonitoringApi"));
B2BMonitoringApi api = (B2BMonitoringApi) bundleContext.getService(serviceRef);
//Create Business Document and set required properties
// Ensue method is createBusinessDocumentCreateEvent to create a Document
BusinessDocumentCreateEvent documentCreateEvent = api.createBusinessDocumentCreateEvent();
documentCreateEvent.setMonitoringReference(headers.get("SAP_MessageProcessingLogID"));
documentCreateEvent.setMonitoringReferenceType(MonitoringReferenceType.MPL);
BusinessDocument document = documentCreateEvent.createBusinessDocument();
def documentId = document.getId();
//Save DocumentId to use in the next steps
message.setProperty("Document_ID", documentId);
document.setSenderMessageType(headers.get("SAP_MessageType"));
document.setReceiverMessageType(headers.get("SAP_MessageType"));
//Hardcoded in this example but can be externalized as a property
document.setSenderAdapterType("HTTP");
document.setReceiverAdapterType("SOAP");
document.setSenderInterchangeControlNumber(headers.get("SAP_ApplicationID"));
document.setReceiverInterchangeControlNumber(headers.get("SAP_ApplicationID"));
document.setSenderTradingPartnerName(message.getProperty("IflowName"));
document.setReceiverTradingPartnerName(headers.get("SAP_MplCorrelationId"));
document.setSenderSystemId(headers.get("SAP_Sender"));
document.setReceiverSystemId(headers.get("SAP_Receiver"));
document.setTransactionDocumentType(BusinessTransactionDocumentType.REQUEST);
document.setProcessingStatus(ProcessingStatus.PROCESSING);
document.setDocumentCreationTimestamp(System.currentTimeMillis());
// Attach Payload to Document
BusinessDocumentPayload inboundPayload = document.createPayload();
inboundPayload.setProcessingPhase(ProcessingPhase.PAYLOAD_PLAIN);
def body = message.getBody(java.lang.String) as String;
inboundPayload.setPayload(body.getBytes(StandardCharsets.UTF_8));
documentCreateEvent.submit();
return message;
}
def Message LogAfterMappingPayload(Message message) {
def headers = message.getHeaders();
def bundleContext = FrameworkUtil.getBundle(Class.forName("com.sap.gateway.ip.core.customdev.util.Message")).getBundleContext();
def serviceRef = bundleContext.getServiceReference(Class.forName("com.sap.it.op.b2b.monitor.api.B2BMonitoringApi"));
B2BMonitoringApi api = (B2BMonitoringApi) bundleContext.getService(serviceRef);
BusinessDocumentSentEvent documentSentEvent = api.createBusinessDocumentSentEvent();
documentSentEvent.setMonitoringReference(headers.get("SAP_MessageProcessingLogID"));
documentSentEvent.setMonitoringReferenceType(MonitoringReferenceType.MPL);
BusinessDocument document = documentSentEvent.createUpdateBusinessDocument(message.getProperty("Document_ID"));
document.setTransactionDocumentType(BusinessTransactionDocumentType.REQUEST);
document.setProcessingStatus(ProcessingStatus.PROCESSING);
document.setDocumentCreationTimestamp(System.currentTimeMillis());
BusinessDocumentPayload inboundPayload = document.createPayload();
inboundPayload.setProcessingPhase(ProcessingPhase.PAYLOAD_PLAIN);
def body = message.getBody(java.lang.String) as String;
inboundPayload.setPayload(body.getBytes(StandardCharsets.UTF_8));
documentSentEvent.submitBeforeMessageTransmission();
return message;
}
def Message LogStatusCompleted(Message message) {
def headers = message.getHeaders();
def bundleContext = FrameworkUtil.getBundle(Class.forName("com.sap.gateway.ip.core.customdev.util.Message")).getBundleContext();
def serviceRef = bundleContext.getServiceReference(Class.forName("com.sap.it.op.b2b.monitor.api.B2BMonitoringApi"));
B2BMonitoringApi api = (B2BMonitoringApi) bundleContext.getService(serviceRef);
BusinessDocumentSentEvent documentSentEvent = api.createBusinessDocumentSentEvent();
documentSentEvent.setMonitoringReference(headers.get("SAP_MessageProcessingLogID"));
documentSentEvent.setMonitoringReferenceType(MonitoringReferenceType.MPL);
BusinessDocument document = documentSentEvent.createUpdateBusinessDocument(message.getProperty("Document_ID"));
document.setProcessingStatus(ProcessingStatus.COMPLETED);
documentSentEvent.submit();
return message;
}
def Message LogStatusError(Message message) {
def headers = message.getHeaders();
// Create event entry in monitoring queue
def bundleContext = FrameworkUtil.getBundle(Class.forName("com.sap.gateway.ip.core.customdev.util.Message")).getBundleContext();
def serviceRef = bundleContext.getServiceReference(Class.forName("com.sap.it.op.b2b.monitor.api.B2BMonitoringApi"));
B2BMonitoringApi api = (B2BMonitoringApi) bundleContext.getService(serviceRef);
ErrorEvent errorEvent = api.createErrorEvent();
errorEvent.setMonitoringReference(message.getHeaders().get("SAP_MessageProcessingLogID"));
errorEvent.setMonitoringReferenceType(MonitoringReferenceType.MPL);
errorEvent.setErrorInformation(message.getProperty("Exception"));
errorEvent.setErrorCategory(ErrorCategory.DOCUMENT_TRANSMISSION);
errorEvent.setTransientError(false);
BusinessDocument document = errorEvent.createUpdateBusinessDocument(message.getProperty("Document_ID"));
document.setProcessingStatus(ProcessingStatus.FAILED);
errorEvent.submit();
return message;
}
def Message LogStatusRetry(Message message) {
def headers = message.getHeaders();
// Create event entry in monitoring queue
def bundleContext = FrameworkUtil.getBundle(Class.forName("com.sap.gateway.ip.core.customdev.util.Message")).getBundleContext();
def serviceRef = bundleContext.getServiceReference(Class.forName("com.sap.it.op.b2b.monitor.api.B2BMonitoringApi"));
B2BMonitoringApi api = (B2BMonitoringApi) bundleContext.getService(serviceRef);
ErrorEvent errorEvent = api.createErrorEvent();
errorEvent.setMonitoringReference(message.getHeaders().get("SAP_MessageProcessingLogID"));
errorEvent.setMonitoringReferenceType(MonitoringReferenceType.MPL);
errorEvent.setErrorInformation(message.getProperty("Exception"));
errorEvent.setErrorCategory(ErrorCategory.DOCUMENT_TRANSMISSION);
errorEvent.setTransientError(false);
BusinessDocument document = errorEvent.createUpdateBusinessDocument(message.getProperty("Document_ID"));
document.setProcessingStatus(ProcessingStatus.RETRY);
errorEvent.submit();
return message;
}
The Details – Sample IFlow showing usage of the script and its script functions
Below is a sample Iflow that uses a HTTP to HTTP Call out. The IFlow receives a Input message, performance a transformation / mapping and sends it to the target system. It has a normal exception handler to get the Exception.
As you would notice in this iflow, we have used 4 Groovy Steps that perform different steps
- Content Modifier : This step type is used to set the Message Headers and Properties. These are described in my previous posts and are standard SAP headers and properties that need no explanation
- Log Incoming Payload : This step is used to Log the Incoming Payload. This step reads the Headers and Properties set in the 1st Content Modifier and Logs the Payload. The scription function used is : LogIncomingPayload
- Log AfterMapping Payload : This step is used to Log the After Mapping Payload. The script function used is : LogAfterMappingPayload
- LogStatusCompleted: This steps is used to Log the Status of the message as Completed. The script function used is : LogStatusCompleted
- The Content Modifier Capture Exception is used to capture the Error in case of a Exception in your process
- Log Status Error : This groovy is used to Log the status of your message as Error. The scription function used is : LogStatusError
The Details – What happens in the script
At a very high level, the script is a extension of the standard reusable scripts from SAP as a part of the Standard Package Delivered by SAP : Cloud Integration – Trading Partner Management V2.
These scripts use SAP Class: com.sap.it.op.b2b.monitor.api.B2BMonitoringApi to generate BusinessDocuments for different Events and pushes the event to CPI. The documentation on the internal working on this is very very sparse ( when I write this in Aug 2023) and using Dark Side of Groovy Scripting: Behind the Scenes of Cloud Platform Integration Runtime as a reference you can download the .jar file from SAP and reference the code. Suffice to say that Internally these scripts are generating events and writing to a Internal Kafka topic.
My own script is leveraging the standard monitoring parameters of CPI. To ensure that I can search for a message using CPI Integration Flow Name and CorrelationID I have used IFlowName in the field : SenderName and CorrelationID in the field ReceiverName.
Final Thoughts
- B2B Monitoring in CPI is much more evolved than A2A Monitoring in the Iflows. B2B Monitoring has native support for Payload Logging. This is an attempt to leverage the B2B Monitoring Capabilities of CPI in A2A Monitoring.
- I see no challenge in using this as is in CPI, as the script is just a adaptation of standard scripts published by SAP, what is unclear is the detailed workings of what happens behind the scenes. So use this at your own risk or get it validated by SAP.
- This has nothing to do with your MPL Logs as B2B logs are stored internally on Kafka (or Event Mesh ..) reverse engineering the implementation classes ( Will updated when more details are available ).
- How long are B2B Messages Persisted? How are they archived? [Updated 31/08/2023]
- Successful messages are persisted for 90 Days
- Failed messages are persisted forever. ( atleast I can see that on my tenant).
- Archival of B2B Messages to BTP DMS as described in this post of mine (SAP CPI – Archive Message Processing Logs (MPLs) to SAP BTP Document Management Service (DMS)) – is still on the roadmap and expected to be released in Q4 2023 as per SAP roadmap explorer here .
- Out of the Box Alerting for failed B2B Messages is also a cool feature that will be available from Q4 2023 as per SAP roadmap explorer here .
- What happens if the event generation fails as this all happens inside a groovy script – All of these are still puzzles that need to be solved. I will update this post when I get the answers to these questions.
- Looking at the B2B Monitoring Capabilities of CPI and the comprehensive solution that is made available by SAP for B2B Monitoring, it is surprising that SAP has not adapted this already for Customers for A2A Monitoring. Most of the pieces of the puzzles are already in place.
- If I had a crystal ball, I would assume that at some point in the future this same feature would be made available for A2A Iflows.
Sample IFlow For Download
If you would like to download this Iflow and try it yourself on your tenant; go for it. Here is the link :Sample IFlow With Script