SAP BTP CPI provides out of the box logging of CPI Message Processing Log to Splunk. This post explores this feature and its current limitations.
Introduction
We have seen how CPI provides multiple options for logging of your message in this post : SAP CPI – Message Monitoring – Standard Out of the Box Features to use in your Iflows. We also also previously explored how MPLs can be Archived to a external DMS : SAP CPI – Archive MPLs to SAP BTP DMS.
Additionally CPI provides you with options to log the MPL ( Message Processing Logs ) to a external Logging solution like Splunk.
Sign up for Splunk Trial
For the purpose of this post, I have set up a Free Trial Splunk to explore this feature. Below are the steps to sign up for a free Splunk and set it up.
- Go to Splunk.com and sign up for a Free Splunk
Set up Index on Splunk
- Login to Splunk with the credentials and URL. If prompted, change your password.
- Navigate to Settings -> Indexes
Set up HEC ( HTTP Event Collector ) on Splunk
- Navigate to Settings -> Data Inputs
- Select Add New on HTTP Event Collector
- Provide a Name : CPI_HEC
- Select index created from previous : cpi_index
Get the HEC Token
- Navigate to Settings – Data Inputs – HTTP Event Collector
- Get the value of the token ( Save it for use in CPI )
Create a Destination : CloudIntegration_MonitoringDataConsumer
Login to your SAP BTP Integration Suite / CPI Subaccount. Create a Destination with the name: CloudIntegration_MonitoringDataConsumer
Parameter | Value |
Name | CloudIntegration_MonitoringDataConsumer |
Type | HTTP |
URL | <<SplunkURL>>:8088 URL from your Login Email from Splunk |
Proxy Type | Internet |
Authentication | No Authentication |
AuthHeader | Splunk <<HEC Token>> HEC Token from previous step |
IndexName | cpi_index Index Name from previous step |
TargetSystem | Splunk |
TrustAll | TRUE |
Create / Update Service Instance
You need to create a Service Instance and Service Key as per Create Service Instance and Service Key
If you already have done that, then you need to update the Service Instance with additional role ExternalLoggingActivate and ExternalLoggingActivationRead roles.
Note the details of the Service Key ( or create a new one ). Parameters to Note in Service Key
- clientid
- clientsecret
- url
- tokenurl
Activate External Logging Settings in Postman
Get Token
- URL : {{tokenurl}}?grant_type=client_credentials
- Authentication : Basic
- User : clientid
- Password: clientsecret
- Get access_token from response
Activate External Logging
- URL : {{url}}/api/v1/activateExternalLogging
- Authentication : Bearer Token
- Use access_token from Previous call
- Method : Post
Your External Logging is now Active
Check External Logging Settings in CPI
Ensure you logout and log in back to your CPI tenant. Once you re-login to your tenant, you will now see the settings in Message Monitoring – Deployed Integration Flows – where External Logging is now enabled. Change the External Logging from None to Info
Test External Logging to Splunk
- Push a Test message to CPI.
- Navigate to Splunk – Search and Reporting
- Search with index=”cpi_index”
Raw Events Sent to Splunk
Successful Events
sourceType = Run
{
"Status": "COMPLETED",
"TenantName": "XXX",
"OriginComponentName": "CPI_XXX",
"Node": "1",
"StartTime": "2023-07-10 12:44:20.919",
"MessageGuid": "AGSr_SQCJ4ELRkRhmC1j3ZjWbajt",
"PreviousComponentName": "CPI_XXX",
"TransmissionLagMillis": 374,
"DurationMillis": 611,
"LogEventId": "7wMzwBJldJdA_7Q7hlZRFA",
"StopTimeMillis": 1688993061530,
"LogLevel": "INFO",
"TransactionId": "bbc923edf9214e0aa3569d5eb6424729",
"StartTimeMillis": 1688993060919,
"LogEventType": "Run",
"LogLevelExternal": "INFO",
"LocalComponentName": "CPI_XXX",
"CustomHeader": {
"po_number": [
"12345"
]
},
"IntegrationFlowId": "Ping",
"CorrelationId": "AGSr_SRB1tWlgkLfjUChzorMcj2n",
"RunId": "AGSr_SS_ysZsBJRfIEQodxE-uI7H",
"LogSentAtMillis": 1688993061904,
"StopTime": "2023-07-10 12:44:21.530",
"ReceiverIds": []
}
sourceType = CustomHeader
{
"LogEventType": "CustomHeader",
"TenantName": "XXX",
"Values": [
"12345"
],
"MessageGuid": "AGSr_SQCJ4ELRkRhmC1j3ZjWbajt",
"RunId": "AGSr_SS_ysZsBJRfIEQodxE-uI7H",
"LogEventId": "7wMzwBJldJdA_7Q7hlZRFA",
"Name": "po_number"
}
Failed Events
{
"OriginComponentName": "CPI_XXX",
"Node": "1",
"MessageGuid": "AGSsAZwlwCAL4Dzf-jMMrJRlz-t2",
"TransmissionLagMillis": 373,
"DurationMillis": 179,
"LogLevel": "INFO",
"StartTimeMillis": 1688994204709,
"LogLevelExternal": "INFO",
"CorrelationId": "AGSsAZztldU6dSwtqTVipkfbSdFo",
"RunId": "AGSsAZxi3Qj78x_Fr_e-Xyx7a0XH",
"LogSentAtMillis": 1688994205261,
"StopTime": "2023-07-10 13:03:24.888",
"Status": "FAILED",
"LastErrorModelStepId": "MessageFlow_7",
"TenantName": "XXX",
"StartTime": "2023-07-10 13:03:24.709",
"PreviousComponentName": "CPI_XXX",
"LogEventId": "utz3Z-Qk72TudcmdaTNY7A",
"StopTimeMillis": 1688994204888,
"TransactionId": "e43d523a17004c4a8afb768567636546",
"LastError": "com.sap.it.rt.adapter.http.api.exception.HttpResponseException: An internal server error occured: bk.requestcatcher.com1: Name or service not known.\nThe MPL ID for the failed message is : AGSsAZwlwCAL4Dzf-jMMrJRlz-t2\n",
"LogEventType": "Run",
"LocalComponentName": "CPI_XXX",
"CustomHeader": {
"po_number": [
"12345"
]
},
"IntegrationFlowId": "Ping",
"ReceiverIds": []
}
Retry and Escalated Events
If you iflow goes into Retry , every Retry if logged as a Event to Splunk. See below example of a Retry and Escalated Event
{
"index": "cpi_index",
"sourcetype": "Run",
"event": {
"OriginComponentName": "CPI_XXX",
"Node": "0",
"MessageGuid": "AGTA2CzpDK-2lFKyl9KQ2DZ6ek3j",
"TransmissionLagMillis": 336,
"DurationMillis": 20,
"LogLevel": "INFO",
"StartTimeMillis": 1690359852074,
"LogLevelExternal": "INFO",
"CorrelationId": "AGTA2Ct3B-eaKoF4e_p7rAI4EH_j",
"RunId": "AGTA2CyKGiPQrzqd4y9MCP9R8-IK",
"LogSentAtMillis": 1690359852430,
"StopTime": "2023-07-26 08:24:12.094",
"Status": "RETRY",
"LastErrorModelStepId": "EndEvent_21",
"TenantName": "XXX",
"StartTime": "2023-07-26 08:24:12.074",
"PreviousComponentName": "CPI_XXX",
"LogEventId": "eQvqbWsdYeQOzD8Shdy8zg",
"StopTimeMillis": 1690359852094,
"TransactionId": "fabab3fe31d34714b56b09d859d4aa4d",
"LastError": "com.sap.esb.camel.error.handler.ErrorEventException: Error Event Exception",
"LogEventType": "Run",
"LocalComponentName": "CPI_XXX",
"IntegrationFlowId": "Ping2",
"ReceiverIds": [
]
}
}
{
"index": "cpi_index",
"sourcetype": "Run",
"event": {
"Status": "ESCALATED",
"TenantName": "XXX",
"OriginComponentName": "CPI_XXX",
"Node": "1",
"StartTime": "2023-07-26 08:29:21.788",
"MessageGuid": "AGTA1zpI00Ir_WK5nJTRveyDcOnN",
"PreviousComponentName": "CPI_XXX",
"TransmissionLagMillis": 324,
"DurationMillis": 17,
"LogEventId": "IechOWvJgoNWpVa-0r59Bg",
"StopTimeMillis": 1690360161805,
"LogLevel": "INFO",
"TransactionId": "c113477101314e9290d39542f6aea5c5",
"StartTimeMillis": 1690360161788,
"LogEventType": "Run",
"LogLevelExternal": "INFO",
"LocalComponentName": "CPI_XXX",
"IntegrationFlowId": "Ping2",
"CorrelationId": "AGTA1zr93MyM0VaziikrSd3wAbzf",
"RunId": "AGTA2WGOnFIeRnY46BxZdETvnpZ9",
"LogSentAtMillis": 1690360162129,
"StopTime": "2023-07-26 08:29:21.805",
"ReceiverIds": [
]
}
}
Check External Logging Status in Postman
To Check External Logging Status use the below API in Postman
- URL : {{url}}/api/v1/ExternalLoggingActivationStatus(‘tenant-name’)
- Leave tenant-name in URL above asis.
- Authentication : Bearer Token
- Use access_token from Previous call
- Method : GET
Deactivate External Logging in Postman
- URL : {{url}}/api/v1/deactivateExternalLogging
- Authentication : Bearer Token
- Use access_token from Previous call
- Method : POST
Final Thoughts
- Activate External Logging to Splunk is very very straight forward and works seamlessly.
- Mesages are logged to Splunk InFlight, i.e, the messages are sent to Splunk Immediately.
- Only MPLs are logged to Splunk.
- Unlike MPL Archival to DMS SAP CPI – Archive MPLs to SAP BTP DMS where Attachments, and Sender and Receiver Channel Messages are logged, in the case of Splunk only MPL is logged.
- ONLY Splunk is supported – i.e, no additional logging systems are supported. No custom Logging System or HTTP Event listener is currently support ( at the time of writing in July 2023 ).
- To understand more what happens under the hook, read this post here: SAP CPI – External Logging – Under the Hood