Error sending points to Wavefront (HTTP -1). Data will be requeued and resent.

This article applies to:

• Data Ingestion

• Product edition: All version

• Feature Category: Application Integration

 

Problem description:

An application is received the below error from the VMware Aria Operations for Applications proxy.

 

11:15:49.359 [wavefrontClientSender-0] WARN c.w.s.common.clients.WavefrontClient - Error sending points to Wavefront (HTTP -1). Data will be requeued and resent.
11:15:49.360 [wavefrontClientSender-0] ERROR c.w.s.c.c.service.ReportingService - Unable to obtain status code from the Wavefront service at http://wavefront-proxy:2878/report?f=wavefront
java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.<init>(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)

 

Note: The preceding log excerpts/Messages are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Error Description: There could be network connectivity issues between the emitting service and the proxy or something else cause java.net.SocketTimeoutException: connect timed out

 

Solution: 

The error is safe to ignore if we're not seeing other exceptions, or a log at WARN level with a message containing "Buffer full, dropping.." The reason why we see this as the data is queued and retried later, at which time it sounds like WavefrontClient doesn't get a timeout when connecting to the WF proxy.

 

Note: To check if metrics are being dropped in your application by WavefrontClient, you can query for ~sdk.java.core.sender.wfclient.points.dropped , with a source of your application's hostname (or some other uniquely identifying tag).

But if we are seeing buffer full, dropping, warning message and exception repeatedly follow the below steps:

•  Try increasing the batch size of their WavefrontClient, which is recommended in the error log about "Buffer full, dropping..."
  The default batch size is 10,000. Try increasing the batch size and see if the issue persists. Instructions for increasing the batch size in the VMware Aria Operations for Applications Java SDK are mentioned here.
• We can also consider increasing the maximum in-memory queue size to something larger than already set (note: the document linked above incorrectly state the default queue size is 50,000). This will give more space/time before the queue fills up and points are dropped.