How Do I Bulk Update Multiple Alerts?

Title: How Do I Bulk Update Multiple Alerts?

This article applies to:

Alerting 

 Product edition: All

 Feature Category: Alerts

 

Overview:

There are times when you may want to update multiple alerts in the same way. One common scenario is updating alert targets. For example, if a user is no longer with the company, you would want to remove that user from any alert notifications. Instead, you may want to point notifications to a different user or team. If the user is listed as a notification target for only a few alerts, it is trivial to update those alerts manually. However, as the number of alerts grows, it becomes less and less easy of a task.

The best practice for handling alert notifications is to use Alert Targets rather than specify specific emails or PagerDuty keys while editing an alert. The reason for this is that by handling the notification target with an Alert Target, you can easily update the Alert Target itself without having to modify each and every alert that uses that Alert Target. 

Nevertheless, this article discusses how you can update multiple alerts.

 

Updating Multiple Alerts:

The primary way to update multiple alerts is to use the API. We will also highlight a customer-created CLI tool based on the Tanzu Observability API that can make things easier. Please note that since this CLI tool is created by a customer, it is not supported by VMware. Any issues or concerns should be directed to the tool's GitHub page.

 

Prerequisites

Identify what needs to be updated across all the alerts of interest. For instance, do you want to update the alert target or do you want to update the condition or the time to fire setting?

Have a process in place to update the field(s) that you want to update.  This can be a script or find and replace with a text editing tool. Just note that the objects will be JSON.

 

Using the Tanzu Observability API Step 1:

Determine which alerts need to be updated. If you do not have this information already, you may need to run a search with the API to extract, at minimum, the alert IDs for the alerts of interest.

To run a search, use the POST /api/v2/search/alert/{facet} API endpoint. Set the facet to id. In the payload, use the query field to specify which alerts to filter for. Valid keys for the query are any of the property keys for an Alert JSON object. If you are unsure of what the available property keys are, use the GET /api/v2/alert endpoint and retrieve just 1 alert to see what the JSON looks like.

 

Example:

In this example, we are retrieving the IDs of all the alerts that have a name containing "prod". Note the need to page through the results to obtain the full set of results. However, notice that in our response we have retrieved a set of IDs.

Request URL: https://CLUSTER.wavefront.com/api/v2/search/alert/id

Payload: 

{
"query": [
{
"key": "name",
"value": "prod",
"matchingMethod": "CONTAINS"
}
]
}

Response:

{ "status": { "result": "OK", "message": "response limited to 10 items for performance", "code": 200 }, "response": { "items": [ "1600096964756", "940000000009", "940000000006", "940000000004", "940000000002", "940000000005", "940000000003", "940000000001" ], "offset": 0, "limit": 10, "totalItems": 8, "moreItems": false } }

 

Using the Tanzu Observability API Step 2:

Retrieve the alerts obtained from Step 1. Use the GET /api/v2/alert/{id} API endpoint to obtain the JSON for each alert. The alert JSON is the data within the "response" field.

Example:

Request URL: https://CLUSTER.wavefront.com/api/v2/alert/1000000012345

Response:

{ "status": { "result": "OK", "message": "", "code": 200 }, "response": { "targetEndpoints": ["target:9R8xByg3rrJ4aiHt"], "modifyAclAccess": true, "hidden": false, "severity": "SEVERE", "minutes": 5, "name": "Example Prod Alert", "id": "1000000012345", "target": "target:9R8xByg3rrJ4aiHt", "status": [ "CHECKING" ], "tags": { "customerTags": [] }, "created": 1000000012345, "updated": 1000000012345, "hostsUsed": [], "orphan": false, "includeObsoleteMetrics": false, "lastQueryTime": 34, "alertType": "CLASSIC", "metricsUsed": [], "evaluateRealtimeData": false, "inTrash": false, "acl": { "canView": [], "canModify": [ "user@domain.com" ] }, "systemOwned": false, "condition": "ts(my.sample.data) < 1", "conditionQBEnabled": false, "displayExpression": "ts(my.sample.data)", "displayExpressionQBEnabled": false, "resolveAfterMinutes": 5, "failingHostLabelPairs": [], "additionalInformation": "", "inMaintenanceHostLabelPairs": [], "activeMaintenanceWindows": [], "updateUserId": "user@domain.com", "prefiringHostLabelPairs": [], "notificants": ["9R8xByg3rrJ4aiHt"], "createUserId": "user@domain.com", "lastProcessedMillis": 1614890662293, "processRateMinutes": 1, "pointsScannedAtLastQuery": 0, "alertsLastDay": 0, "alertsLastWeek": 0, "alertsLastMonth": 0, "numPointsInFailureFrame": 0, "creatorId": "user@domain.com", "updaterId": "user@domain.com", "createdEpochMillis": 1000000012345, "updatedEpochMillis": 1000000012345, "deleted": false, "targetInfo": [], "failingHostLabelPairLinks": [], "sortAttr": 840, "severityList": [ "SEVERE" ] } }

 


Using the Tanzu Observability API Step 3:

Update the alert JSON accordingly. 

 

Using the Tanzu Observability API Step 4:

Update each alert with the updated JSON. Use the PUT /api/v2/alert/{id} endpoint and include the updated alert JSON as the payload. After this is completed, you've successfully updated the alerts.

 

Example:
In this example, the alert target was updated. Notice in the response that the alert target IDs have updated.

 

Request URL: https://CLUSTER.wavefront.com/api/v2/alert/1000000012345

Payload: updated JSON from Step 3

Response:

{ "status": { "result": "OK", "message": "", "code": 200 }, "response": { "targetEndpoints": ["target:12345yg3rrJ4aiHt"], "modifyAclAccess": true, "hidden": false, "severity": "SEVERE", "minutes": 5, "name": "Example Prod Alert", "id": "1000000012345", "target": "target:12345yg3rrJ4aiHt", "status": [ "CHECKING" ], "tags": { "customerTags": [] }, "created": 1000000012345, "updated": 1000000098765, "hostsUsed": [], "orphan": false, "includeObsoleteMetrics": false, "lastQueryTime": 34, "alertType": "CLASSIC", "metricsUsed": [], "evaluateRealtimeData": false, "inTrash": false, "acl": { "canView": [], "canModify": [ "user@domain.com" ] }, "systemOwned": false, "condition": "ts(my.sample.data) < 1", "conditionQBEnabled": false, "displayExpression": "ts(my.sample.data)", "displayExpressionQBEnabled": false, "resolveAfterMinutes": 5, "failingHostLabelPairs": [], "additionalInformation": "", "inMaintenanceHostLabelPairs": [], "activeMaintenanceWindows": [], "updateUserId": "user@domain.com", "prefiringHostLabelPairs": [], "notificants": ["12345yg3rrJ4aiHt"], "createUserId": "user@domain.com", "lastProcessedMillis": 1614890662293, "processRateMinutes": 1, "pointsScannedAtLastQuery": 0, "alertsLastDay": 0, "alertsLastWeek": 0, "alertsLastMonth": 0, "numPointsInFailureFrame": 0, "creatorId": "user@domain.com", "updaterId": "user@domain.com", "createdEpochMillis": 1000000012345, "updatedEpochMillis": 1000000098765, "deleted": false, "targetInfo": [], "failingHostLabelPairLinks": [], "sortAttr": 840, "severityList": [ "SEVERE" ] } }

 

Using the CLI Tool:

The CLI Tool makes the same underlying calls to the Tanzu Observability API as described above but simplifies the process for the end user. 

 

The equivalent for the Step 1 and Step 2 examples above is:

wf alert search name~prod -f json -M > alert.json

This will export the alert JSON for all the alerts that contain "prod" in their names.

 

After making the appropriate changes to the alert JSON and assuming alert-updated.json contains the updated JSON, the equivalent for Step 4 above is:

wf alert import alert-updated.json

 

See also:

If both update and migration need to happen for alerts, use what is described in this article along with what is described in Migrating Objects Between Tanzu Observability Environments.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk