Proxy errors writing to buffer when using persistent volumes. – VMware Aria Operations for Applications Knowledge Base

This article applies to:

• Integration Proxy

• Product edition: All Versions

• Feature Category: Integrations

Problem Description:

WF-005: Error requesting exclusive access to the buffer lock file
WF-1: CRITICAL (Losing data): WF-1: Error adding task to the queue:

Cause:
When VMware Aria Operations for Application is receiving more metrics, ts() or hs(), then it can send the proxy will buffer the data while waiting to resend. When using persistent volumes the correct permissions must be given within the pod. Errors codes WF-005 & WF-1 indicate a problem writing to the buffer.

Note: Writing to buffer errors can also be caused by space issues. See Proxy CRITICAL Messages link below.

Error Messages/Messages:

ERROR [TaskQueueFactoryImpl:createTaskQueue] WF-005: Error requesting exclusive access to the buffer lock file /var/spool/wavefront-proxy/buffer.points.2878.0.lck - please make sure that no other processes access this file and restart the proxy
ERROR [AbstractDataSubmissionTask:log] [2878] CRITICAL (Losing data): WF-1: Error adding task to the queue: Storage queue is not available!

Verification:

Check the buffer location referenced in the WF-005 error to see if the directory has been created.

If it has NOT been created this is possible permission issue.
If it has been created look for any space issues. See Proxy Critical Messages link below.

If no buffer files exist, check proxy logs during startup for any file permission errors.

Hint: If proxy logs have rolled restart the pod and then immediately pull logs for any startup errors.

Example of errors during startup:

chown: cannot read directory '/var/spool/wavefront-proxy/lost+found': Permission denied
chown: changing ownership of '/var/spool/wavefront-proxy': Operation not permitted

Resolution:
This is not an issue with VMware's Aria Operations for Application proxy but rather a permission issue with the Kubernetes pod, as the proxy cannot write to the persistent volume.

Please review your custom yaml file for proxy deployment to make sure permissions are set correctly.

Here is a link to our default proxy helm template, proxy-deployment.yaml for your reference.
Here also is a Kubernetes document for your review on configuring your custom yaml for pod configuration. Configure a Security Context for a Pod or Container

Its recommended to use the new Wavefront Kubernetes Operator which sets up the proxy / proxy buffer directory as a Kubernetes emptyDir with the correct permissions thus avoiding the node file system permission issues

Comments