Contact us
, ,

SAP CPI – External Logging to Splunk and Current Limitations

Bhavesh Kantilal

By.

Bhavesh Kantilal

•

min read

Share

Post Views: 1,197

SAP BTP CPI provides out of the box logging of CPI Message Processing Log to Splunk. This post explores this feature and its current limitations.

Introduction

We have seen how CPI provides multiple options for logging of your message in this post : SAP CPI – Message Monitoring – Standard Out of the Box Features to use in your Iflows. We also also previously explored how MPLs can be Archived to a external DMS : SAP CPI – Archive MPLs to SAP BTP DMS.

Additionally CPI provides you with options to log the MPL ( Message Processing Logs ) to a external Logging solution like Splunk.

Sign up for Splunk Trial

For the purpose of this post, I have set up a Free Trial Splunk to explore this feature. Below are the steps to sign up for a free Splunk and set it up.

Set up Index on Splunk

  • Login to Splunk with the credentials and URL. If prompted, change your password.
  • Navigate to Settings -> Indexes

Set up HEC ( HTTP Event Collector ) on Splunk

  • Navigate to Settings -> Data Inputs
  • Select Add New on HTTP Event Collector
  • Provide a Name : CPI_HEC
  • Select index created from previous : cpi_index

Get the HEC Token

  • Navigate to Settings – Data Inputs – HTTP Event Collector
  • Get the value of the token ( Save it for use in CPI )

Create a Destination : CloudIntegration_MonitoringDataConsumer

Login to your SAP BTP Integration Suite / CPI Subaccount. Create a Destination with the name: CloudIntegration_MonitoringDataConsumer

ParameterValue
NameCloudIntegration_MonitoringDataConsumer
TypeHTTP
URL<<SplunkURL>>:8088
URL from your Login Email from Splunk
Proxy TypeInternet
AuthenticationNo Authentication
AuthHeaderSplunk <<HEC Token>>
HEC Token from previous step
IndexNamecpi_index
Index Name from previous step
TargetSystemSplunk
TrustAllTRUE
Destination Configuration

Create / Update Service Instance

You need to create a Service Instance and Service Key as per Create Service Instance and Service Key

If you already have done that, then you need to update the Service Instance with additional role ExternalLoggingActivate and ExternalLoggingActivationRead roles.

Note the details of the Service Key ( or create a new one ). Parameters to Note in Service Key

  • clientid
  • clientsecret
  • url
  • tokenurl

Activate External Logging Settings in Postman

Get Token

  • URL : {{tokenurl}}?grant_type=client_credentials
  • Authentication : Basic
    • User : clientid
    • Password: clientsecret
  • Get access_token from response

Activate External Logging

  • URL : {{url}}/api/v1/activateExternalLogging
  • Authentication : Bearer Token
    • Use access_token from Previous call
  • Method : Post

Your External Logging is now Active

Check External Logging Settings in CPI

Ensure you logout and log in back to your CPI tenant. Once you re-login to your tenant, you will now see the settings in Message Monitoring – Deployed Integration Flows – where External Logging is now enabled. Change the External Logging from None to Info

Test External Logging to Splunk

  • Push a Test message to CPI.
  • Navigate to Splunk – Search and Reporting
  • Search with index=”cpi_index”

Raw Events Sent to Splunk

Successful Events

sourceType = Run

{
	"Status": "COMPLETED",
	"TenantName": "XXX",
	"OriginComponentName": "CPI_XXX",
	"Node": "1",
	"StartTime": "2023-07-10 12:44:20.919",
	"MessageGuid": "AGSr_SQCJ4ELRkRhmC1j3ZjWbajt",
	"PreviousComponentName": "CPI_XXX",
	"TransmissionLagMillis": 374,
	"DurationMillis": 611,
	"LogEventId": "7wMzwBJldJdA_7Q7hlZRFA",
	"StopTimeMillis": 1688993061530,
	"LogLevel": "INFO",
	"TransactionId": "bbc923edf9214e0aa3569d5eb6424729",
	"StartTimeMillis": 1688993060919,
	"LogEventType": "Run",
	"LogLevelExternal": "INFO",
	"LocalComponentName": "CPI_XXX",
	"CustomHeader": {
		"po_number": [
			"12345"
		]
	},
	"IntegrationFlowId": "Ping",
	"CorrelationId": "AGSr_SRB1tWlgkLfjUChzorMcj2n",
	"RunId": "AGSr_SS_ysZsBJRfIEQodxE-uI7H",
	"LogSentAtMillis": 1688993061904,
	"StopTime": "2023-07-10 12:44:21.530",
	"ReceiverIds": []
}

sourceType = CustomHeader

{
	"LogEventType": "CustomHeader",
	"TenantName": "XXX",
	"Values": [
		"12345"
	],
	"MessageGuid": "AGSr_SQCJ4ELRkRhmC1j3ZjWbajt",
	"RunId": "AGSr_SS_ysZsBJRfIEQodxE-uI7H",
	"LogEventId": "7wMzwBJldJdA_7Q7hlZRFA",
	"Name": "po_number"
}
Failed Events
{
	"OriginComponentName": "CPI_XXX",
	"Node": "1",
	"MessageGuid": "AGSsAZwlwCAL4Dzf-jMMrJRlz-t2",
	"TransmissionLagMillis": 373,
	"DurationMillis": 179,
	"LogLevel": "INFO",
	"StartTimeMillis": 1688994204709,
	"LogLevelExternal": "INFO",
	"CorrelationId": "AGSsAZztldU6dSwtqTVipkfbSdFo",
	"RunId": "AGSsAZxi3Qj78x_Fr_e-Xyx7a0XH",
	"LogSentAtMillis": 1688994205261,
	"StopTime": "2023-07-10 13:03:24.888",
	"Status": "FAILED",
	"LastErrorModelStepId": "MessageFlow_7",
	"TenantName": "XXX",
	"StartTime": "2023-07-10 13:03:24.709",
	"PreviousComponentName": "CPI_XXX",
	"LogEventId": "utz3Z-Qk72TudcmdaTNY7A",
	"StopTimeMillis": 1688994204888,
	"TransactionId": "e43d523a17004c4a8afb768567636546",
	"LastError": "com.sap.it.rt.adapter.http.api.exception.HttpResponseException: An internal server error occured: bk.requestcatcher.com1: Name or service not known.\nThe MPL ID for the failed message is : AGSsAZwlwCAL4Dzf-jMMrJRlz-t2\n",
	"LogEventType": "Run",
	"LocalComponentName": "CPI_XXX",
	"CustomHeader": {
		"po_number": [
			"12345"
		]
	},
	"IntegrationFlowId": "Ping",
	"ReceiverIds": []
}

Retry and Escalated Events

If you iflow goes into Retry , every Retry if logged as a Event to Splunk. See below example of a Retry and Escalated Event

{
    "index": "cpi_index",
    "sourcetype": "Run",
    "event": {
        "OriginComponentName": "CPI_XXX",
        "Node": "0",
        "MessageGuid": "AGTA2CzpDK-2lFKyl9KQ2DZ6ek3j",
        "TransmissionLagMillis": 336,
        "DurationMillis": 20,
        "LogLevel": "INFO",
        "StartTimeMillis": 1690359852074,
        "LogLevelExternal": "INFO",
        "CorrelationId": "AGTA2Ct3B-eaKoF4e_p7rAI4EH_j",
        "RunId": "AGTA2CyKGiPQrzqd4y9MCP9R8-IK",
        "LogSentAtMillis": 1690359852430,
        "StopTime": "2023-07-26 08:24:12.094",
        "Status": "RETRY",
        "LastErrorModelStepId": "EndEvent_21",
        "TenantName": "XXX",
        "StartTime": "2023-07-26 08:24:12.074",
        "PreviousComponentName": "CPI_XXX",
        "LogEventId": "eQvqbWsdYeQOzD8Shdy8zg",
        "StopTimeMillis": 1690359852094,
        "TransactionId": "fabab3fe31d34714b56b09d859d4aa4d",
        "LastError": "com.sap.esb.camel.error.handler.ErrorEventException: Error Event Exception",
        "LogEventType": "Run",
        "LocalComponentName": "CPI_XXX",
        "IntegrationFlowId": "Ping2",
        "ReceiverIds": [
            
        ]
    }
}

{
    "index": "cpi_index",
    "sourcetype": "Run",
    "event": {
        "Status": "ESCALATED",
        "TenantName": "XXX",
        "OriginComponentName": "CPI_XXX",
        "Node": "1",
        "StartTime": "2023-07-26 08:29:21.788",
        "MessageGuid": "AGTA1zpI00Ir_WK5nJTRveyDcOnN",
        "PreviousComponentName": "CPI_XXX",
        "TransmissionLagMillis": 324,
        "DurationMillis": 17,
        "LogEventId": "IechOWvJgoNWpVa-0r59Bg",
        "StopTimeMillis": 1690360161805,
        "LogLevel": "INFO",
        "TransactionId": "c113477101314e9290d39542f6aea5c5",
        "StartTimeMillis": 1690360161788,
        "LogEventType": "Run",
        "LogLevelExternal": "INFO",
        "LocalComponentName": "CPI_XXX",
        "IntegrationFlowId": "Ping2",
        "CorrelationId": "AGTA1zr93MyM0VaziikrSd3wAbzf",
        "RunId": "AGTA2WGOnFIeRnY46BxZdETvnpZ9",
        "LogSentAtMillis": 1690360162129,
        "StopTime": "2023-07-26 08:29:21.805",
        "ReceiverIds": [
            
        ]
    }
}

Check External Logging Status in Postman

To Check External Logging Status use the below API in Postman

  • URL : {{url}}/api/v1/ExternalLoggingActivationStatus(‘tenant-name’)
    • Leave tenant-name in URL above asis.
  • Authentication : Bearer Token
    • Use access_token from Previous call
  • Method : GET

Deactivate External Logging in Postman

  • URL : {{url}}/api/v1/deactivateExternalLogging
  • Authentication : Bearer Token
    • Use access_token from Previous call
  • Method : POST

Final Thoughts

  • Activate External Logging to Splunk is very very straight forward and works seamlessly.
  • Mesages are logged to Splunk InFlight, i.e, the messages are sent to Splunk Immediately.
  • Only MPLs are logged to Splunk.
  • Unlike MPL Archival to DMS SAP CPI – Archive MPLs to SAP BTP DMS where Attachments, and Sender and Receiver Channel Messages are logged, in the case of Splunk only MPL is logged.
  • ONLY Splunk is supported – i.e, no additional logging systems are supported. No custom Logging System or HTTP Event listener is currently support ( at the time of writing in July 2023 ).
  • To understand more what happens under the hook, read this post here: SAP CPI – External Logging – Under the Hood
, ,

Bhavesh Kantilal