Validating metrics received at the proxy

This article applies to:

Data Ingestion

Product edition: Proxy Current Release

Feature Category: Proxy

Overview

When troubleshooting issues with metric data it may be necessary to check that the points are received at the proxy in the format intended.
To verify the proxy is receiving the metric data and the metrics are in the correct format, enable raw valid point logging. This logs all the valid points received at the proxy to a separate file.

 

** Caution this will use disk space quickly, and should be only enabled for a short period of time while troubleshooting. **

 

Valid Point Logging:

To log all valid points separately in a single file, follow these steps:

Open the <wavefront_config_path>/log4j2.xml configuration file and uncomment the corresponding sections below, for example /etc/wavefront/wavefront-proxy/log4j2.xml in a typical Linux install. 

 <Appenders>

<!-- Uncomment the RollingFile section below to log all valid points to a file -->

<RollingFile name="ValidPointsFile" fileName="${log-path}/wavefront-valid-points.log"
filePattern="${log-path}/wavefront-valid-points-%d{yyyy-MM-dd}-%i.log" >
<PatternLayout>
<pattern>%m%n</pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy interval="1"/>
<SizeBasedTriggeringPolicy size="1024 MB"/>
</Policies>
<DefaultRolloverStrategy max="10">
<Delete basePath="${log-path}" maxDepth="1">
<IfFileName glob="wavefront-valid*.log" />
<IfLastModified age="7d" />
</Delete>
</DefaultRolloverStrategy>
</RollingFile>

</Appenders>
 <Loggers>

<!-- Uncomment AppenderRef and set level="ALL" to log all valid points to a file -->
<AsyncLogger name="RawValidPoints" level="ALL" additivity="false">
<AppenderRef ref="ValidPointsFile"/>
</AsyncLogger>

</Loggers>

 

By default, valid point entries are logged to the <wavefront_log_path>/wavefront-valid-points.log file, for example /var/log/wavefront/wavefront-valid-points.log in a typical Linux install.

The <wavefront_log_path>/wavefront.log file should indicate that the logging is enabled as shown below:

2021-04-02 05:53:24,436 INFO [sampling:refreshLoggerState] Valid points logging is now enabled with 100.0% sampling

If the metric points are not written to the wavefront-valid-points.log file, verify that the data is not being blocked by logging all the raw blocked data in a separate file for analysis. 

 

Blocked Point Logging:

To log the blocked data separately in its own file, follow these steps:

Open the <wavefront_config_path>/log4j2.xml configuration file and uncomment the corresponding sections below, for example /etc/wavefront/wavefront-proxy/log4j2.xml in a typical Linux install.

 <Appenders>

<!-- Uncomment the RollingFile section below to log blocked points to a file -->

<RollingFile name="BlockedPointsFile" fileName="${log-path}/wavefront-blocked-points.log"
filePattern="${log-path}/wavefront-blocked-points-%d{yyyy-MM-dd}-%i.log" >
<PatternLayout>
<pattern>%m%n</pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy interval="1"/>
<SizeBasedTriggeringPolicy size="100 MB"/>
</Policies>
<DefaultRolloverStrategy max="10">
<Delete basePath="${log-path}" maxDepth="1">
<IfFileName glob="wavefront-blocked*.log" />
<IfLastModified age="31d" />
</Delete>
</DefaultRolloverStrategy>
</RollingFile>

</Appenders>
 <Loggers>
<AsyncLogger name="RawBlockPoints" level="WARN" additivity="false">
<AppenderRef ref="BlockedPointsFile" />
</AsyncLogger>
</Loggers>

Set level="WARN" will log only rejected points, setting level="INFO" will log points filtered out by allow/block preprocessor rules.
By default, blocked point entries are logged to the <wavefront_log_path>/wavefront-blocked-points.log file and the log file is rolled over every day when its size reaches 100MB.

 

See Proxy log4j2.xml reference for additional information on logging points from Jaeger tracing spans and Zipkin tracing spans. 

For additional information, please also see the following:
Wavefront data format best practices.

 

 

 

 

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk