When to use raw vs non-raw aggregation

Aggregating data is a use case shared across all customers, but data shape is not. For example, identifying the average amount of resources consumed (E.g. disk, memory) within a specific set of data may be common. The volume of reporting sources, frequency of reporting, and/or the uniformity of reporting, such as whether all reporting sources report at the same interval or if skipped intervals are expected, are typically unique to each scenario.  

Understanding your use case and identifying data shape will determine whether raw (E.g. rawsum()) or non-raw aggregation (E.g. sum()) is right for you. 

 

Query Examples

raw aggregation: rawsum(ts("test.application.error"))
non-raw aggregation: sum(ts("test.application.error"))

 

What's the difference?

Both raw and non-raw aggregation methods accomplish the same goal: Consolidating several time series into 1 or more trends in order to quickly evaluate behavior. However, they take differing paths (and thus different results in many cases) to come to an answer. The heart of this difference can be attributed to interpolation.

After applying a non-raw aggregation function to a query, Wavefront will execute the process of aggregating data at any interval where at least 1 truly reported value exists. In Wavefront, interpolation is the process of generating a made-up data value for 1 or more time series where they don't exist, and can only occur between two truly reported values within a series. For example, if you are aggregating 3 time series and only 1 of those series reported a value at 1pm, then Wavefront will attempt to generate an interpolated value for the other 2 time series at 1pm prior to aggregation. While a performance impact likely won't be observed up to a certain point, keep in mind that the process of interpolation does require additional resources to be consumed in order to return a result. This can ultimately result in long query load times for heavier data sets and/or larger time windows.  

In contrast to non-raw aggregation, raw aggregation functions will never attempt to generate an interpolated value when aggregating 2 or more time series. 

mceclip1.png

  

Which aggregation approach is right for me?

Your use case and data shape should dictate which aggregation approach you select. Luckily, there are some context clues that we can use to help guide you!

Let's start with understanding your data shape. How many time series are you wanting to aggregate together (10's, 1000's)? Given that interpolation requires additional resources in order to render a result, using a non-raw aggregation function on several thousands of time series may come at the cost of query performance. In this scenario, that trade-off may help you to determine which aggregation approach. Specially if you are attempting to aggregate those thousands of time series over several weeks or months worth of data.

Also consider whether skipped reporting intervals are common and what you'd like to see in those scenarios. For example, imagine you are looking to aggregate the number of errors occurring across multiple time series. In this case, it's important to note how you report no errors occurring. Do you report a zero when no errors occur, or does it simply skip the reporting interval where there were no errors? The former would typically not be impacted by interpolation associated with non-raw aggregation since there's a zero value present (and thus no need for an interpolated value to be generated). The latter could theoretically be impacted by non-raw aggregation because an interpolated data value could be generated for a time series when no error actually occurred. Understanding your reporting intervals and the resulting aggregation behavior should help in making your aggregation approach.

This doesn't mean that non-raw aggregation methods should be avoided. In fact, they can provide quick value and results whenever your reporting intervals are staggered. For example, imagine you are evaluating 10 time series over a 1 hour window. Each time series reports once per minute, but they don't align (some are reported on the minute boundary while others may report a few seconds after). In this scenario, a non-raw aggregation function can be extremely useful in providing you an expected trend result without the need for aligning your data. This simplicity can be very helpful! With that said, let's reconsider this scenario but replace 10 time series with 1000 time series -and- 1 hour with a 1 week chart window. The reporting frequency is still once per minute, but a larger data set like this may result in there being at least 1 true value present at each second boundary. In this case, we'd need to generate 100's of interpolated data values every second of every minute for an entire week window in order to generate an expected trend result. In this case, the query performance impact may be undesired. If so, then you'd need to align your data to 1-minute boundaries and then from there consider whether interpolated values are required (E.g. non-raw aggregation) or not (raw aggregation).

Ex. rawavg(align(1m,ts("my.data")))

 

Summary & Additional Resources

In conclusion, Wavefront offers both raw and non-raw aggregation functions for a variety of use cases. If you need to avoid interpolation from negatively impacting your results, or you believe query performance will be impacted, then consider using raw aggregation functions. If interpolation will not negatively reflect your results based on your use case, then non-raw aggregation functions can be helpful.

If Wavefront documentation and/or the information included in this article did not answer your questions, then please reach out to Support for help. Additionally, the following docs could be helpful:

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk