This article applies to:
- Troubleshooting/Data Ingestion
- Feature Category: Telegraf Agent (Third Party Integration)
When sending metrics from a Telegraf agent through a load balancer to a Tanzu Observability proxy, for example from an AWS EC2 instance passing through an AWS ELB to a Tanzu Observability Proxy, over a socket connection, some broken connection messages are observed in the logs.
See below for example:
telegraf[xxx]: 2021-03-03T09:25:19Z E! [agent] Error writing to outputs.wavefront: Wavefront sending error: write tcp 192.168.xxx.xxx:36948->192.168.xxx.xx:2878: write: broken pipe
telegraf[xxx]: 2020-03-03T09:26:10Z I! connected to Wavefront proxy at address: 192.168.xxx.xx:2878
The Tanzu Observability output plugin, when communicating over a socket connection assumes a stable connection between the Telegraf agent and the Tanzu Observability proxy. The stable connection requirement is not satisfied if for example, a load balancer resets the connection after a defined period of time such as 60 seconds etc.
Use the HTTP protocol instead of the socket connection, as it doesn't require a stable connection.
To change the Tanzu Observability output plugin to use "http" mode, specify the "url" and "port" in the output.wavefront telegraf plugin.
[[outputs.wavefront]] url = "http://192.168.xxx.xxx:2878" metric_separator = "." source_override = ["hostname", "agent_host", "node_host"] convert_paths = true
Alternatively, increase the load balancer idle connection timeout value, for example 120 seconds etc.
Note: The preceding log excerpts/Messages are only examples. Date, time, and environmental variables may vary depending on your environment.