[INTERNAL] Proxy errors writing to buffer when using persistent volumes.

This article applies to:

• Integration Proxy

• Product edition: All Versions

• Feature Category: Integrations


Problem Description:

WF-005: Error requesting exclusive access to the buffer lock file
WF-1: CRITICAL (Losing data): WF-1: Error adding task to the queue:


Cause:
When VMware Aria Operations for Application is receiving more metrics, ts() or hs(), then it can send the proxy will buffer the data while waiting to resend.   When using persistent volumes the correct permissions must be given within the pod. Errors codes WF-005 & WF-1 indicate a problem writing to the buffer.  

Note: Writing to buffer errors can also be caused by space issues.  See Proxy CRITICAL Messages link below.

 

Error Messages/Messages:

  • ERROR [TaskQueueFactoryImpl:createTaskQueue] WF-005: Error requesting exclusive access to the buffer lock file /var/spool/wavefront-proxy/buffer.points.2878.0.lck - please make sure that no other processes access this file and restart the proxy
  • ERROR [AbstractDataSubmissionTask:log] [2878] CRITICAL (Losing data): WF-1: Error adding task to the queue: Storage queue is not available!

Verification:

Check the buffer location referenced in the WF-005 error to see if the directory has been created.

  • If it has NOT been created this is possible permission issue.
  • If it has been created look for any space issues. See Proxy Critical Messages link below.

If no buffer files exist, check proxy logs during startup for any file permission errors.

Hint: If proxy logs have rolled restart the pod and then immediately pull logs for any startup errors.

 

Example of errors during startup:

chown: cannot read directory '/var/spool/wavefront-proxy/lost+found': Permission denied
chown: changing ownership of '/var/spool/wavefront-proxy': Operation not permitted

 

Resolution:
This is not an issue with VMware's Aria Operations for Application proxy but rather a permission issue with the Kubernetes pod, as the proxy cannot write to the persistent volume. 

Please review your custom yaml file for proxy deployment to make sure permissions are set correctly.

Its recommended to use the new Wavefront Kubernetes Operator which sets up the proxy  / proxy buffer directory as a Kubernetes emptyDir with the correct permissions thus avoiding the node file system permission issues

 

See also:

Proxy Troubleshooting - Proxy CRITICAL Messages 
[INTERNAL] Common Proxy Error Logs for TO Logs
[INTERNAL] Common Tanzu Observability by Wavefront Proxy Log Messages
Default proxy helm template

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk