Alert Reference

Alert Name Group Severity Description Action
EnvoyWafTooManySecurityEvents Security Events Virtual-Host major Virtual Host WAF security events detected. Consider blocking the relevant users/IPs using FastACL or Network Policy or Service Policy.
FluentbitRetriesFailed Log Collection Error Infrastructure critical Log collection has failed to forward logs to RE site for more than 15 minutes. Check network connectivity between CE and RE site.
KubeAPILatencyHigh K8S API Error Infrastructure minor Kubernetes API latency at 99th percentile is too high for more than 2 seconds. Possible iterminent problem which may occur during parallel application updates. Check HW utilization of CE site. If persist for longer than hour contant support.
KubeClientCertificateExpiration K8S Client Certificate Error Infrastructure minor Kubernetes certificates is expiring for your Volterra Site. In order to avoid interruption, upgrade to latest available Volterra Software Version. Upgrade Volterra Software Version to latest available.
KubeClientCertificateExpiration K8S Client Certificate Error Infrastructure major Kubernetes certificates is expiring for your Volterra Site. In order to avoid interruption, upgrade to latest available Volterra Software Version. Upgrade Volterra Software Version to latest available.
KubeClusterCPUOvercommit K8S Cluster CPU Overcommit Infrastructure minor Site has overcommitted CPU requests for Pods, failure may cause Site disruption. Increase capacity by adding a Node or Reduce Pod workload.
KubeClusterMemOvercommit K8S Cluster Memory Overcommit Infrastructure minor Site has overcommitted RAM memory resource requests for Pods and cannot tolerate any node failure. Add new node into site or deprovision workload.
KubeCronJobRunning K8S Job Too Long IaaS-CaaS minor Kubernetes CronJob running for more than hour. Job can be stuck or it is expected to run longer. Check logs from Kubernetes Pod. Contact support in case of non-customer vk8 workload.
KubeDaemonSetRolloutStuck K8S Daemonset Error IaaS-CaaS minor Kubernetes DaemoSet desired Pods are not scheduled or ready. Check Kubernetes Pod status, events and logs in vK8s cluster. Contact support in case of non vk8s DaemonSet.
KubeDeploymentGenerationMismatch K8S Deployment Error IaaS-CaaS minor Deployment generation does not match, this indicates that the Deployment has failed but has not been rolled back. Check Kubernetes Pod status, events and logs in vK8s cluster. Contact support in case of non vk8s Deployment.
KubeJobFailed K8S Job Failed IaaS-CaaS minor Kubernetes Job failed to complete in last 2 hours. Check Kubernetes Job and Pod status, events and logs in vK8s cluster. Contact support in case of etcd job.
KubeNodeNotReady K8S Node Error Infrastructure critical Site node has been unready for more than 1 hr. Pods cannot be scheduled or deprovisioned since node is not responding. Check Node and HW status in console UI. Reboot node. If persistent for longer than 1 hr contact support.
KubeNodeTooManyPods K8S Node Error Infrastructure minor Number of pods running near maximum. Add new node into site or deprovision workload.
KubePersistentVolumeSpaceLow K8S PVC Error IaaS-CaaS minor Kubernetes PersistentVolumeClaim is getting out of space. Resize PVC or clean disk.
KubePodCPUThrottlingHigh K8S Pod CPU Throttled Infrastructure major Kubernetes Pod container is throttling it's CPU limits. Increase flavor for vk8s Deployment or StatefulSet definition. Contact support in case of non vk8s Pod.
KubePodContainerTooMuchMemory IaaS-CaaS critical More than 90% of allowed memory is being used by container. Add more replicas.
KubePodCrashLooping K8S Pod Crashing IaaS-CaaS minor Kubernetes Pod container restarting often. Possible causes can be out of memory limit (OOM), liveness probe or container entrypoint failure. Check Kubernetes Pod status, events and logs in vK8s cluster. Contact support in case of non vk8s Deployment.
KubePodNotReady K8S Pod Not Ready IaaS-CaaS minor Pod has been in a non-ready state for more than 10 min. The reason might be readiness probe failures, scheduling due out of quotas or broken node. Check Kubernetes Pod status, events and logs in vK8s cluster. Contact support in case of non vk8s Deployment.
KubeStatefulSetReplicasMismatch K8S StatefulSet Error IaaS-CaaS minor Kubernetes StatefulSet has not matched the expected number of Pod replicas for longer than 15 minutes. Check Kubernetes Pod status, events and logs in vK8s cluster. Contact support in case of non vk8s Deployment.
KubeVersionMismatch K8S Internal Error Infrastructure minor There are different versions of Kubernetes components running. This can be caused by failure during Volterra Software Upgrade. Check Volterra Software Upgrade status. Ignore if upgrade is in progress.
NodeFilesystemFilesFillingUp Filesystem runs out of files Infrastructure critical Filesystem at node is predicted to run out of files within the next 8 hours. Check disk usage at Site dashboard. Deprovision workload or add new node into site.
NodeFilesystemOutOfFiles Node Filesystem Error Infrastructure minor Filesystem at node has only a few percent available inodes left. Check disk usage at Site dashboard. Deprovision workload or add new node into site. Do disk resize in case of cloud CE. Contact support in case problem persist.
NodeFilesystemSpaceFillingUp Node Filesystem Error Infrastructure minor Filesystem at node is predicted to run out of space within the next 24 hrs. Check disk usage at Site dashboard. Deprovision workload or add new node into site. Do disk resize in case of cloud CE.
NodeLoadHigh Node Load High Infrastructure minor Node has higher load than 1 per CPU for more than 10 mins. Add new node into site or deprovision workload.
ServiceClientErrorPerSourceSite Virtual Host Client Error Virtual-Host major More than 10% of the requests from site to service failed due to client error. Some clients are sending invalid requests to the virtual-host. Consider blocking the relevant users/IPs using Volterra Policy features.
ServiceEndpointHealthcheckFailure Endpoint healthcheck failure Virtual-Host minor Healthcheck failed for virtual-host endpoint. Check the health of the origin servers. Check connectivity of origin servers to Volterra.
ServiceServerErrorPerSourceSite Virtual Host Server Error Virtual-Host major ServiceServerErrorPerSourceSite Proxy is seeing excessive errors from upstream origin servers. Check the health of the origin servers. Check connectivity of origin servers to Volterra.
SiteCustomerTunnelInterfaceDown Customer Tunnel Interface Down Infrastructure major Connection from CE to a single RE is down. Some functionality will be limited. Check physical and network connectivity of the CE.
SitePhysicalInterfaceDown Physical Interface Down Infrastructure critical One of the physical interfaces of CE went down. Check physical and network connectivity of the CE.
SiteTunnelInterfaceDown Tunnel Interface Down Infrastructure critical Connection from both REs to CE are down. Majority of functionality will be impacted. Check physical and network connectivity of the CE
VesMauriceSiteNodeHeartbeatMissed Site Heartbeat Down Infrastructure major Node at site did not send heartbeat for more than 20 minutes. Check network connectivity and power status of node in Site. If running, trying rebooting the node.
VesMauriceSiteUpgradeFailing Site Upgrade Failing Infrastructure critical Volterra software upgrade is failing at Site. It retries every 10 minutes and keeps updating the status. Check Volterra Software status message info. Contact support if problem persist for more than 30 minutes.
VesVoltShareDecryptionError VoltShare Decryption Error VoltShare major Decrypt operation has failures. Check secret policy or admin policy. -