Troubleshooting alert notifications from Tanzu Observability to Big panda webhook targets

This article applies to:

Product edition: Current

Feature Category: Alerts /Alert targets


Overview
:

Customers may experience circumstances when Tanzu Observability alert payloads are not being received at the Big Panda integration side. Big Panda notifications may have also appeared to have been snoozed allowing for outages to not be properly alerted upon.


This article would describe and help you to investigate and troubleshoot the different alerting error scenarios and modify them with all the remediation steps.

 

Common Configuration:

A typical configuration seen with alert targets is by utilising webhooks. It is common to see Tanzu Observability configured to have Big Panda alert targets, using the webhook feature as shown below , 

mceclip1.png

 

Frequent issues seen with Bigpanda webhook alert targets are these alert firing within Tanzu Observability but not with the Big Panda side of the alert target.

This KB outlines the troubleshooting steps for the target notifications.

Note: The preceding log excerpts/Messages are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Procedure:

With the outlined configuration we have now encountered a problem. Tanzu Observability is functioning correctly and firing for a defined alert, however the defined webhook target is not alerting in conjunction with Tanzu Observability as desired. This is a frustrating scenario as we can have dependencies /workflows outside of Tanzu Observability that we rely upon for other functions such as notifications.

 

Items to Validate : 

1. The first step would be to validate alert "Triggers" field in the edit alert target page to include

You can also check for other options as per your target trigger scenario
details to be read at - Customising alert target 
mceclip1.png

 

2. Change the Alert Target "Type" from Webhook to an Email to see the exact data that is being sent in the Body Template section in [Edit Alert Target]

The https://webhook.site/ is testing site for troubleshooting webhooks and mustache template as you can see in real time what is coming in with your alert target notifications.

It is also possible to have a separate alert with the same template as in the defined alert to test with webhook and get more visibility into what is happening for debugging purposes. 

                                                                 

3. Check the mustache template is catching every scenario. There might be a case that only one of  failingAlertSeries, newlyFailingAlertSeries or  recoveredAlertSeries would be set in the template.

Make sure that all the necessary fields are added and validate the notifications with webhook

Please see the template format in our documentation at - Alert Template

 

4. Validate if you see any events on the Events page which could be related to Bigpanda alert failure. The event description would provide a descriptive idea of what could have gone wrong.

 

5. In the edit Alert target page, make sure the Content-Type is selected to "application/json" 

 

6. One reason could be due to the configuration of the WaveFront alert target and the included authorization headers. The {secret_value} is an obfuscation and should not be directly copied to the new target.

 

7. Another frequent issue identified and observed is an alert that fired and resolved within Tanzu Observability and appeared in Big Panda however, did not resolve. This creates problem as your external alert targets workflows might be dependent on alert resolving notifications

The reason for notifications being missed/ alerts not resolving on BigPanda is due to these specific alerts being edited within Tanzu Observability while firing/ongoing.

Tanzu Observability already has an improvement request for fixing this use case.

 

If all of the above steps are validated and still the issue persists, please take next steps:

  1. Raise a case with Bigpanda / validate if they are receiving the payload and whether it has been suppressed by them 
  2. Engage Tanzu Observability Support. Read more at How to Engage Technical Support

                                   

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk