Contact us
, , , , ,

SAP CPI – External Logging (To Splunk)- Under the Hood

Bhavesh Kantilal

By.

Bhavesh Kantilal

•

min read

Share

Post Views: 2,399

Introduction

We had a look at SAP CPI ‘s External Logging to Splunk in this post : SAP CPI – External Logging to Splunk and Current Limitations. What happens under the hood? Lets take a look.

SAP’s External Logging to Splunk feature on CPI is a new feature where we have seen that Message Processing Logs – MPLs are Sent to Splunk immediately on the fly. At the point of writing this post, this feature is only support for Splunk.

What if you have a external logging system ( Cloud / OnPremise) where you want these Logs to be made available. Theoretically, this should be a easy solution considering all we need is a HTTP Event Listener ( or webhook or any HTTP End point). SAP has already defined a JSON schema for your MPLs. You should be able to receive these MPLs to any HTTP Event listener – well atleast thats what I thought when I started looking under the hood.

Initial Idea

I have already shown the JSON that is received by Splunk from CPI in my post SAP CPI – External Logging to Splunk and Current Limitations. Ofcourse this just the JSON event to Splunk but the headers etc of the JSON Message are not visible. So, my initial idea was to change the Destination CloudIntegration_MonitoringDataConsumer created in the SAP BTP Cockpit of your Integration Subaccount to point to a HTTP Event listener or HTTP iflow running on CPI. The idea was I could just change the HTTP end point , point it to CPI and then capture the raw HTTP event.

Below is my CPI Iflow that is now in a position to retrieve this event with a simple HTTP Adapter

I changed the destination CloudIntegration_MonitoringDataConsumer to point to this iflow:

When I tried to push my message with External Logging enabled; I was expecting that the External Log will push this message to this Iflow but the message was still logging to Splunk!

Learning#1 – Change in Destination Name or Values need reactivation of External Logging

Even thought I had changed the Logging destination URL and credentials, the message were still logging to Splunk, hence the learning that any changes to this Destination : CloudIntegration_MonitoringDataConsumer required a Reactivation of external Logging.

So, I decided to re-activate External Logging by de-activating it and then activating it again. I have explained the steps in my previous post to Activate External Logging.

Deactivate External Logging – Successful

Activate External Logging Again with Destination Containing CPI HTTP EndPoint – Fails

When I tried to Activate the external logging again with the Destination containing the CPI endpoint, I got the error as below screenshot. The error was that OPTIONS end point was being called on /services/collector/event. 

{
    "result": "ERROR",
    "message": "Error during HTTP call: XXX.it-cpi023-rt.cfapps.eu20-001.hana.ondemand.com/http/ExternalLoggingEvent: Name or service not known",
    "infos": [
        {
            "description": "Read Tenant Information",
            "result": "SUCCESS",
            "dataSent": "",
            "dataReceived": ""
        },
        {
            "description": "Validating Connection Settings",
            "result": "SUCCESS",
            "dataSent": "",
            "dataReceived": "All Settings Valid"
        },
        {
            "description": "Instantiating Selected Adapter",
            "result": "SUCCESS",
            "dataSent": "",
            "dataReceived": "Adapter Created"
        },
        {
            "description": "Calling OPTIONS on endpoint",
            "result": "ERROR",
            "dataSent": "OPTIONS /services/collector/event",
            "dataReceived": "Error during HTTP call: XXX.it-cpi023-rt.cfapps.eu20-001.hana.ondemand.com/http/ExternalLoggingEvent: Name or service not known"
        }
    ]
}

Learning#2 – CPI when trying to log to Splunk or a External Logging System – adds a end point : /services/collector/event

Even though our CPI end point is Ok, internally in the code of CPI there is validation when you activate external logging to Splunk where it adds /services/collector/event to the endpoint.

Ofcourse in CPI we know that HTTP endpoints always have a /http/ in it, so listening to these requests on CPI was not a option.

Use requestcatcher to catch the HTTP Request

As an alternate I decided to use https://requestcatcher.com/ and add my own endpoint here to allow me to see if that works.

I changed my destination to now point to my requestcatcher endpoint https://sprintegrate.requestcatcher.com/

Now when I tried to activate the External Logging this was successful and I could see the raw request in requestcatcher.com.

A HTTP POST and OPTIONS called was being done on /services/collector/event, the default HTTP Event Listener end point for Splunk.

Learning#3 – Any Custom Listener should listen on services /collector/event

As we have seen in previous section, SAP CPI for external logging to Splunk acutomatically adds /services/collector/event to the HTTP end point. So any custom event listener you build in your landscape be should run on /services/collector/event

I decided to change my endpoint on CPI with a wildcard * but the External Logging Activation still failed when pointing to CPI. The error was the same as the error in previous section- Error during HTTP call: XXX.it-cpi023-rt.cfapps.eu20-001.hana.ondemand.com/http/ExternalLoggingEvent: Name or service not known

Learning#4 – The Endpoint cannot have any extra parts other than hostname.

As we have seen when I pushed the request to requestcatcher.com vs the CPI endpoint, the destination has some hard rule. The URL for your custom HTTP Listener should only contain the hostname as https://hostname and the URL should be https://hostname/services/collector/event .

The current implementation from SAP as of writing this post ( July 2023) does not allow you to add any additional parts to the URL.

SAP APIM to the rescue

To enable me to overcome the limitation that logging is only possible over Splunk and not to a external system due to the URL limitation, as a PoC I ended up creating a SAP APIM Proxy that listens in on the endpoint

https://SAPAPIM/services/collector/event and sends this to CPI to see the Raw Event.

See screenshots below of the API and the Destination on BTP

Raw Event JSON Generated by CPI

After using SAP APIM as a proxy, I was able to get the event from SAP APIM and look at the raw Event. I forwarded this from SAP APIM to a CPI Iflow.

JSON String is Escaped

  • When I looked at the JSON String in SAP APIM / CPI, I realised that the JSON is escaped. See sample below
{
	"index": "cpi_index",
	"sourcetype": "Run",
	"event": "{\"Status\":\"COMPLETED\",\"TenantName\":\"XXX\",\"OriginComponentName\":\"CPI_XXX\",\"Node\":\"1\",\"StartTime\":\"2023-07-20 07:32:06.221\",\"MessageGuid\":\"AGS44vbFQi1MshUSV0bm6MitbvOb\",\"PeviousComponentName\":\"CPI_XXX\",\"TransmissionLagMillis\":122,\"DurationMillis\":800,\"LogEventId\":\"6Ti12mVAkSgutAKElR2vnw\",\"StopTimeMillis\":1689838327021,\"LogLevel\":\"INFO\",\"TransactionId\":\"17e7af6214d745ab9cd66970c6c75e77\",\"StartTimeMillis\":1689838326221,\"LogEventType\":\"Run\",\"LogLevelExternal\":\"INFO\",\"LocalComponentName\":\"CPI_XXX\",\"CustomHeader\":{\"po_number\":[\"12345\"]},\"IntegrationFlowId\":\"Ping2\",\"CorrelationId\":\"AGS44vai3SKMZOnux3FwwQLgvClS\",\"RunId\":\"AGS44vYeRXS60TME8emQXS2Cgqf_\",\"LogSentAtMillis\":1689838327143,\"StopTime\":\"2023-07-20 07:32:07.021\",\"ReceiverIds\":[]}"
}
  • I used the below groovy in CPI then to unescaped the JSON
import com.sap.gateway.ip.core.customdev.util.Message
import groovy.json.JsonOutput
import groovy.json.JsonSlurper
import org.apache.commons.text.StringEscapeUtils

def Message processData(Message message) {
  def jsonString = message.getBody(java.lang.String) as String
  def jsonSlurper = new JsonSlurper()
  def object = jsonSlurper.parseText(jsonString)
  //object.event = jsonSlurper.parseText(object.event.replace('\\\\', '\\'))
  object.event = jsonSlurper.parseText(StringEscapeUtils.unescapeJson(object.event))
  message.setBody(JsonOutput.prettyPrint(JsonOutput.toJson(object)))
  return message
}

Escaped JSON

{
    "index": "cpi_index",
    "sourcetype": "Run",
    "event": {
        "Status": "COMPLETED",
        "TenantName": "XXX",
        "OriginComponentName": "CPI_XXX",
        "Node": "1",
        "StartTime": "2023-07-20 07:38:29.382",
        "MessageGuid": "AGS45HUOcmtJRShOwnKj2IzJ2NvK",
        "PreviousComponentName": "CPI_XXX",
        "TransmissionLagMillis": 440,
        "DurationMillis": 632,
        "LogEventId": "XqjsFEriAY5l_QNzalpRZg",
        "StopTimeMillis": 1689838710014,
        "LogLevel": "INFO",
        "TransactionId": "4a07f462fb3e499d8993001b13ffcce3",
        "StartTimeMillis": 1689838709382,
        "LogEventType": "Run",
        "LogLevelExternal": "INFO",
        "LocalComponentName": "CPI_XXX",
        "CustomHeader": {
            "po_number": [
                "12345"
            ]
        },
        "IntegrationFlowId": "Ping2",
        "CorrelationId": "AGS45HVgwEcnF9asdI9p4vhWqQme",
        "RunId": "AGS45HU0aoJ76-qagQPgoqbCoxcV",
        "LogSentAtMillis": 1689838710454,
        "StopTime": "2023-07-20 07:38:30.014",
        "ReceiverIds": [
            
        ]
    }
}

Understanding the Raw JSON Event generated by CPI

Santhosh has blogged here Splunk – Part 2 : SAP CPI MPL Logging on how Splunk HEC Events work but I have also explained how the CPI Splunk logs match what Splunk expects.

Final Thoughts

  • SAP for External Logging has some hard rules defined. At this moment only Splunk is supported.
  • Even if you want to configure your own Custom HTTP event listener in any other system ( other than Splunk), your HTTP Listener should be on https://hostname/services/collector/event
  • If you can get a HTTP event listener running on this end point https://hostname/services/collector/event; then you can easily retrieve the HTTP Events that are meant for Splunk to any Logging System. In my case I used APIM as a Proxy to get the requests.
  • While SAP does not officially support any other External Logging System ( yet as of July 2023), using SAP APIM or any other HTTP Event Listeners as a proxy over https://hostname/services/collector/event as a end point you can get the events logged to any System you want.
  • Attachments and raw payloads are not supported yet ( Unfortunate and hopefully comes soon ).
  • If you want Attachments and Sender Channel Payload – only option is to use MPL Archiving to SAP BTP DMS as described here as an Out of the box Solution : SAP CPI – Archive Message Processing Logs (MPLs) to SAP BTP Document Management Service (DMS)
, , , ,

Bhavesh Kantilal