This article applies to:
• Product edition: Current
• Feature Category: Alerts /Alert targets
Overview:
Customers may experience circumstances when VMware Aria Operations for Applications alert payloads are not being received at the Big Panda integration side. Big Panda notifications may have also appeared to have been snoozed allowing for outages to not be properly alerted upon.
This article would describe and help you to investigate and troubleshoot the different alerting error scenarios and modify them with all the remediation steps.
Common Configuration:
A typical configuration seen with alert targets is by utilising webhooks. It is common to see VMware Aria Operations for Applications configured to have Big Panda alert targets, using the webhook feature as shown below ,
Frequent issues seen with Bigpanda webhook alert targets are these alert firing within VMware Aria Operations for Applications but not with the Big Panda side of the alert target.
This KB outlines the troubleshooting steps for the target notifications.
Note: The preceding log excerpts/Messages are only examples. Date, time, and environmental variables may vary depending on your environment.
Procedure:
With the outlined configuration we have now encountered a problem. VMware Aria Operations for Applications is functioning correctly and firing for a defined alert, however the defined webhook target is not alerting in conjunction with VMware Aria Operations for Applications as desired. This is a frustrating scenario as we can have dependencies /workflows outside of VMware Aria Operations for Applications that we rely upon for other functions such as notifications.
Items to Validate :
1. The first step would be to validate alert "Triggers" field in the edit alert target page to include
2. Change the Alert Target "Type" from Webhook to an Email to see the exact data that is being sent in the Body Template section in [Edit Alert Target]
The https://webhook.site/ is testing site for troubleshooting webhooks and mustache template as you can see in real time what is coming in with your alert target notifications.
It is also possible to have a separate alert with the same template as in the defined alert to test with webhook and get more visibility into what is happening for debugging purposes.
3. Check the mustache template is catching every scenario. There might be a case that only one of failingAlertSeries, newlyFailingAlertSeries or recoveredAlertSeries would be set in the template.
Make sure that all the necessary fields are added and validate the notifications with webhook
Please see the template format in our documentation at - Alert Template
4. Validate if you see any events on the Events page which could be related to Bigpanda alert failure. The event description would provide a descriptive idea of what could have gone wrong.
5. In the edit Alert target page, make sure the Content-Type is selected to "application/json"
6. One reason could be due to the configuration of the VMware Aria Operations for Applications alert target and the included authorization headers. The {secret_value} is an obfuscation and should not be directly copied to the new target.
7. Another frequent issue identified and observed is an alert that fired and resolved within VMware Aria Operations for Applications and appeared in Big Panda however, did not resolve. This creates problem as your external alert targets workflows might be dependent on alert resolving notifications
The reason for notifications being missed/ alerts not resolving on BigPanda is due to these specific alerts being edited within VMware Aria Operations for Applications while firing/ongoing.
VMware Aria Operations for Applications already has an improvement request for fixing this use case.
If all of the above steps are validated and still the issue persists, please take next steps:
- Raise a case with Bigpanda / validate if they are receiving the payload and whether it has been suppressed by them
- Engage VMware Aria Operations for Applications. Read more at How to Engage Technical Support
Comments