Telegraf connection errors when using load balancer with Tanzu Observability proxy

This article applies to:

  • Troubleshooting/Data Ingestion
  • Feature Category: Telegraf Agent (Third Party Integration)

Problem Description:

When sending metrics from a Telegraf agent through a load balancer to a Tanzu Observability proxy, for example from an AWS EC2 instance passing through an AWS ELB to a Tanzu Observability Proxy, over a socket connection, some broken connection messages are observed in the logs.

See below for example: 

telegraf[xxx]: 2021-03-03T09:25:19Z E! [agent] Error writing to outputs.wavefront: Wavefront sending error: write tcp 192.168.xxx.xxx:36948->192.168.xxx.xx:2878: write: broken pipe
telegraf[xxx]: 2020-03-03T09:26:10Z I! connected to Wavefront proxy at address: 192.168.xxx.xx:2878

 

Cause:

The Tanzu Observability output plugin, when communicating over a socket connection assumes a stable connection between the Telegraf agent and the Tanzu Observability proxy. The stable connection requirement is not satisfied if for example, a load balancer resets the connection after a defined period of time such as 60 seconds etc.

 

Workaround:

Use the HTTP protocol instead of the socket connection, as it doesn't require a stable connection.
To c
hange the Tanzu Observability output plugin to use "http" mode, specify the "url" and "port" in the output.wavefront telegraf plugin.
For example:

[[outputs.wavefront]]
  url = "http://192.168.xxx.xxx:2878"
  metric_separator = "."
  source_override = ["hostname", "agent_host", "node_host"]
  convert_paths = true

See also:

Tanzu Observability Output Plugin 

Telegraf Configuration Windows

Telegraf Configuration Linux

Telegraf Configuration Mac

 

Alternatively, increase the load balancer idle connection timeout value, for example 120 seconds etc. 

 

Note: The preceding log excerpts/Messages are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk