How to monitor time Proxy is spending with preprocessing rules



This article applies to:

Data Ingestion

Feature Category: Proxy


Out of the box, the Wavefront Proxy emits metrics to ensure adopters are able to understand how it is performing. Metrics emitted paint of picture of the profile of metric types, points, connections, JVM health, version, uptime, health and preprocessor processing.  

The preprocessor processing is what we will spend time focusing on in this article. Why is monitoring the preprocessor useful? How much clock time the JVM is spending on each rule and on all rules, in addition to being able to determine the overall effectiveness of the rule(s).

Rules that are not optimized can contribute to data lag, meaning a delay in Wavefront receiving the data in a timely manner.


Out of the Box Dashboards:

In the 'Proxy Troubleshooting' section, there are several dashboards available to monitor the inner-workings of a proxy.  The time the proxy is taking processing the rules, can impact the delivery of data to Wavefront.  The metrics, '~proxy.points.*.received.lag.p95', help report the 95th percentile of time differences (in milliseconds) between the timestamp on a point and the time that the proxy received it. Large numbers indicate backfilling old data, or clock drift in the sending systems. You can also graph other percentiles.

Preprocessor rules: CPU time per proxy

This chart paints an aggregate view how long each proxy is spending executing all of the preprocessing rules.  

Preprocessor rules: CPU time per rule 

This chart paints an aggregate view across all proxies showing how much time it is taking executing each rule for each message.  This chart will help you display outliers and identify preprocessing rules which should be optimized.

Preprocessor rules: hit ratio, %

This chart will help you identify if there are preprocessing rules that are no longer in use or impact a high number of metrics being ingested.  This should be used to identify if there are some rules which should be deprecated or possibly tuned.

In conclusion:

When creating preprocessing rules, it is important to understand the impact of performance to the proxy to ensure optimal delivery of data to Wavefront.  Please review the Regex Notes to ensure the expression leverages Best Practices.  And as always, please ensure your proxies are on the latest release.


See also:

Monitoring Wavefront Proxies

Proxy Preprocessor Rules

Preprocessor Rule Conditions

Proxy Troubleshooting

Proxy Deployment Options

Advanced Proxy Configuration

Install and Manage Wavefront Proxies

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request


Powered by Zendesk