This article describes commonly seen messages in the Tanzu Observability by Wavefront Proxy logs. Messages are categorized by severity level:
- Proxy Start-up Messages
- Preprocessor Rule Messages
- Proxy Check-in
- Processed Since Start
- Proxy Queue Size
- WF-300 Cannot Parse
Proxy Start-up Messages
INFO [proxy:parseArguments] Wavefront Proxy version <Proxy version>, runtime: OpenJDK Runtime Environment (Azul Systems, Inc.) <JDK version>
This should be the first start-up message and indicates the version of the Tanzu Observability Proxy.
INFO [proxy:lambda$startListeners$7] listening on port: 2878 for Wavefront metrics
This message indicates that the Proxy has been configured to listen for Tanzu Observability data formatted metrics on port 2878. If tracing or histograms are configured, you should see similar corresponding messages. For example,
INFO [proxy:startTraceListener] listening on port: 30000 for trace data
INFO [proxy:run] setup complete
This message indicates that Proxy setup has completed.
Preprocessor Rule Messages
INFO [PreprocessorConfigManager:loadFromStream] Loaded <#> rules for port :: <port>
This message indicates that a certain number of rules have been detected and loaded for the specified port
INFO [proxy:checkin] Checking in: https://<cluster>.wavefront.com/api
This message indicates that the Proxy has been able to check in with the Tanzu Observability backend. These messages should appear at approximately one minute intervals.
Processed Since Start
INFO [AbstractReportableEntityHandler:printTotal] [<port>] <Points/Histograms/Spans/SpanLogs/etc> processed since start: <#>; blocked: <#>
This message indicates the number of data points processed and blocked for the specified type on the specified port since the start of the Proxy process.
Proxy Queue Size
INFO [QueuedAgentService:lambda$new$5] current retry queue sizes: [<#>/<#>/<#>/<#>]
This message indicates the size of the Proxy's queues.
WF-300 Cannot Parse
INFO [AbstractReportableEntityHandler:reject] [<port>] blocked input: [WF-300 Cannot parse: "<data point>"; remote: <address> [<port>]; reason: "Could not parse: <data point>", root cause: "<portion of data point> is not allowed here!"
This message indicates that the data point is not formatted properly and was thus, blocked by the Proxy.
Confirm that the data point conforms to the format configured for the specified port. The Proxy can handle various different data formats. Ensure that the data format you've configured for the port matches the format of data arriving at that port.
If the port is configured for standard Tanzu Observability data format, see the documentation for the proper syntax to cross-check the data point mentioned in the log message.
406 - Cannot Post Push Data
WARN [QueuedAgentService:handleTaskRetry] Cannot post push data result to Wavefront servers.
Will enqueue and retry later: java.util.concurrent.RejectedExecutionException: Response not accepted by server: 406
This indicates that the Proxy has queued up some data and will retry momentarily. This behavior is very common and often occurs when a burst of data is attempted to be sent to the backend. Since, the burst of data is at a higher momentary rate than the backend limit (based on your contract rate), the Proxy queues up the data momentarily to help "smooth" out your data rate. Typically, this is not a cause for concern.
Check if the Proxy's queue/backlog size is staying at the same size or growing in the Tanzu Observability Service and Proxy Data dashboard > Proxies: overview section. If it is, your attempted rate of ingest is higher than allowed by the backend limit. Either lower the rate of data arriving at the Proxies or contact Support to request a higher backend limit. Note that you may incur overage charges if your overall rate of data ingest is higher than your contract rate.
If the Proxy's queue size is spiky (going up and coming down, close to 0), then the Proxy is effectively smoothing out bursts in your rate of data ingest. This is normal behavior and is not a cause for concern.
Global Rate Limit Exceeded
WARN [QueuedAgentService:run] [RETRY THREAD <#>] Wavefront server rejected the submission (global rate limit exceeded) - will attempt later.
This message indicates that the Proxy attempted to send data to the backend but the overall ingest rate has exceeded the backend limit. As a result, the Proxy will continue to keep the data queued. This message is often seen along with the 406 message.
The current rate of data is higher than the backend limit and data will stay in the queues to be retried momentarily. See 406 - Cannot Post Push Data for more details.
Disconnected, Unterminated String
WARN [PlainTextOrHttpFrameDecoder:decodeLast] Client <client address> [<port>] disconnected, leaving unterminated string. Input (<#> bytes) discarded: "<discarded data>"
This message indicates that the Proxy has received a data point without a line terminator (ie. a newline character). This usually happens when the client disconnects prematurely, leaving the Proxy with an incomplete (partially sent) data point.
Ensure that data points are terminated by a newline character.
Check on the connectivity between the client and the Proxy. Switching to the HTTP protocol for sending data to the Proxy will also help.
HTTP 401 Unauthorized
2021-02-18 22:52:28,376 ERROR [proxy:checkinError] HTTP 401 Unauthorized: Please verify that your server and token settings are correct and that the token has Proxy Management permission!
This message indicates the Tanzu Observability Proxy is unable to check in with the the token provided. The token or token's account may have been deleted or may not have the required permissions.
- Validate the token the Tanzu Observability Proxy is attempting to use is correct and active for the User Account or Service Account.
- Ensure the User Account or Service Account that the token the proxy is attempting to check in with has the Proxies permission.
WF-1 - Queues Full
ERROR [QueuedAgentService:addTaskToSmallestQueue] CRITICAL (Losing points!): WF-1: Submission queue is full.
com.squareup.tape.FileException: Failed to add entry.
CRITICAL (Losing data): WF-1: Error adding task to the queue:
This indicates that the Proxy is running out of disk space or the queue (buffer) files are exceeding their maximum allowed size (normally 2GB).
Ensure that the Proxy has enough disk space. Check the size of the buffer files (located at the spool path). If the files are at or near the maximum allowed size, data is likely arriving at the Proxy at a higher rate than the backend limit. Decrease the rate of data arriving at the Proxy or distribute your data ingest to another Proxy.