Common Tanzu Observability by Wavefront Proxy Log Messages

This article describes commonly seen messages in the Tanzu Observability by Wavefront Proxy logs. Messages are categorized by severity level:

INFO

WARN

ERROR

CRITICAL

 

INFO

 

Proxy Start-up Messages

Message

INFO [proxy:parseArguments] Wavefront Proxy version <Proxy version>, runtime: OpenJDK Runtime Environment (Azul Systems, Inc.) <JDK version>

Explanation

This should be the first start-up message and indicates the version of the Tanzu Observability Proxy.

 

Message:

INFO [proxy:lambda$startListeners$7] listening on port: 2878 for Wavefront metrics

Explanation:

This message indicates that the Proxy has been configured to listen for Tanzu Observability data formatted metrics on port 2878. If tracing or histograms are configured, you should see similar corresponding messages. For example, 

INFO [proxy:startTraceListener] listening on port: 30000 for trace data

 

Message:

INFO [proxy:run] setup complete

Explanation:

This message indicates that Proxy setup has completed.

 

 

Preprocessor Rule Messages

Message:

INFO [PreprocessorConfigManager:loadFromStream] Loaded <#> rules for port :: <port>

Explanation:

This message indicates that a certain number of rules have been detected and loaded for the specified port

 

 

Proxy Check-in

Message:

INFO [proxy:checkin] Checking in: https://<cluster>.wavefront.com/api

Explanation:

This message indicates that the Proxy has been able to check in with the Tanzu Observability backend. These messages should appear at approximately one minute intervals.

 

 

Processed Since Start

Message:

INFO [AbstractReportableEntityHandler:printTotal] [<port>] <Points/Histograms/Spans/SpanLogs/etc> processed since start: <#>; blocked: <#> 

Explanation:

This message indicates the number of data points processed and blocked for the specified type on the specified port since the start of the Proxy process.

 

 

Proxy Queue Size

Message:

INFO  [QueuedAgentService:lambda$new$5] current retry queue sizes: [<#>/<#>/<#>/<#>]

Explanation:

This message indicates the size of the Proxy's queues.

 

 

WF-300 Cannot Parse

Message:

INFO [AbstractReportableEntityHandler:reject] [<port>] blocked input: [WF-300 Cannot parse: "<data point>"; remote: <address> [<port>]; reason: "Could not parse: <data point>", root cause: "<portion of data point> is not allowed here!"

Explanation:

This message indicates that the data point is not formatted properly and was thus, blocked by the Proxy.

Potential Resolution:

Confirm that the data point conforms to the format configured for the specified port. The Proxy can handle various different data formats. Ensure that the data format you've configured for the port matches the format of data arriving at that port.

If the port is configured for standard Tanzu Observability data format, see the documentation for the proper syntax to cross-check the data point mentioned in the log message.

 

 

WARN

 

406 - Cannot Post Push Data

Message:

WARN  [QueuedAgentService:handleTaskRetry] Cannot post push data result to Wavefront servers.
 Will enqueue and retry later: java.util.concurrent.RejectedExecutionException: Response not accepted by server: 406

Explanation:

This indicates that the Proxy has queued up some data and will retry momentarily. This behavior is very common and often occurs when a burst of data is attempted to be sent to the backend. Since, the burst of data is at a higher momentary rate than the backend limit (based on your contract rate), the Proxy queues up the data momentarily to help "smooth" out your data rate. Typically, this is not a cause for concern.

Potential Resolution:

Check if the Proxy's queue/backlog size is staying at the same size or growing in the Tanzu Observability Service and Proxy Data dashboard > Proxies: overview section. If it is, your attempted rate of ingest is higher than allowed by the backend limit. Either lower the rate of data arriving at the Proxies or contact Support to request a higher backend limit. Note that you may incur overage charges if your overall rate of data ingest is higher than your contract rate.

If the Proxy's queue size is spiky (going up and coming down, close to 0), then the Proxy is effectively smoothing out bursts in your rate of data ingest. This is normal behavior and is not a cause for concern.

 

 

Global Rate Limit Exceeded

Message:

WARN  [QueuedAgentService:run] [RETRY THREAD <#>] Wavefront server rejected the submission (global rate limit exceeded) - will attempt later.

Explanation:

This message indicates that the Proxy attempted to send data to the backend but the overall ingest rate has exceeded the backend limit. As a result, the Proxy will continue to keep the data queued. This message is often seen along with the 406 message. 

Potential Resolution:

The current rate of data is higher than the backend limit and data will stay in the queues to be retried momentarily. See 406 - Cannot Post Push Data for more details.

 

 

Disconnected, Unterminated String

Message:

WARN [PlainTextOrHttpFrameDecoder:decodeLast] Client <client address> [<port>] disconnected, leaving unterminated string. Input (<#> bytes) discarded: "<discarded data>"

Explanation:

This message indicates that the Proxy has received a data point without a line terminator (ie. a newline character). This usually happens when the client disconnects prematurely, leaving the Proxy with an incomplete (partially sent) data point. 

Potential Resolution:

Ensure that data points are terminated by a newline character.
Check on the connectivity between the client and the Proxy. Switching to the HTTP protocol for sending data to the Proxy will also help.

 

ERROR

 

HTTP 401 Unauthorized

Message:

2021-02-18 22:52:28,376 ERROR [proxy:checkinError] HTTP 401 Unauthorized: Please verify that your server and token settings are correct and that the token has Proxy Management permission!

Explanation:

This message indicates the Tanzu Observability Proxy is unable to check in with the the token provided. The token or token's account may have been deleted or may not have the required permissions.

Potential Resolution:

CRITICAL

 

WF-1 - Queues Full

Message:

ERROR [QueuedAgentService:addTaskToSmallestQueue] CRITICAL (Losing points!): WF-1: Submission queue is full.
com.squareup.tape.FileException: Failed to add entry.

or

CRITICAL (Losing data): WF-1: Error adding task to the queue:

Explanation:

This indicates that the Proxy is running out of disk space or the queue (buffer) files are exceeding their maximum allowed size (normally 2GB). 

Potential Resolution:

Ensure that the Proxy has enough disk space. Check the size of the buffer files (located at the spool path). If the files are at or near the maximum allowed size, data is likely arriving at the Proxy at a higher rate than the backend limit. Decrease the rate of data arriving at the Proxy or distribute your data ingest to another Proxy.

 

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk