EKS Node Monitoring Agent

The EKS Node Monitoring Agent detects health issues on Amazon EKS worker nodes by parsing system logs and surfacing status information through Kubernetes NodeConditions. When paired with Amazon EKS node auto repair, detected issues can trigger automatic node replacement or reboot.

For detailed configuration options and usage documentation, refer to the Amazon EKS Node Health documentation.

Overview

The agent runs as a DaemonSet on each node and monitors for issues across several categories:

• Kernel - Process limits, kernel bugs, soft lockups
• Networking - VPC CNI (IPAMD) issues, interface problems, connectivity
• Storage - EBS throughput/IOPS limits, I/O delays
• Container Runtime - Pod termination issues, probe failures
• Accelerated Hardware - NVIDIA GPU errors (XID codes), AWS Neuron issues, DCGM diagnostics

For each category, the agent applies a dedicated NodeCondition to worker nodes (e.g., KernelReady, NetworkingReady, StorageReady, AcceleratedHardwareReady). These conditions integrate with Amazon EKS node auto repair to automatically remediate unhealthy nodes.

Project Layout

.
├── api/                    # API definitions and CRDs
├── charts/                 # Helm chart for deployment
├── cmd/                    # Application entry point
├── examples/               # Integration examples
├── hack/                   # Build and utility scripts
├── monitors/               # Health monitoring plugins
├── pkg/                    # Core packages
└── test/                   # Integration tests

Installation

It is recommended to install the EKS Node Health Monitoring Agent as an EKS add-on. For Helm installation instructions, see charts/eks-node-monitoring-agent/README.md.

For detailed configuration options and usage documentation, refer to the Amazon EKS Node Health documentation.

Configuring Monitors

By default all monitors are enabled. Individual monitors can be disabled via the Helm chart’s nodeAgent.monitors configuration or by providing a config file at /etc/nma/config.yaml.

Helm Values

Each monitor supports enabled: true/false to enable or disable it:

nodeAgent:
  monitors:
    networking:
      enabled: false
    neuron:
      enabled: false

The networking monitor additionally supports allowedIPTablesChains to suppress UnexpectedRejectRule warnings for rules in custom chains. Entries must use table/chain format:

nodeAgent:
  monitors:
    networking:
      allowedIPTablesChains:
        - "filter/MY-CUSTOM-CHAIN"

The networking monitor also supports excludedInterfaceNameRegexps to suppress InterfaceNotUp / InterfaceNotRunning findings for interfaces that are not part of Kubernetes node networking. This is useful on accelerated instance types (e.g. P6) that expose host-visible Mellanox/NVIDIA IPoIB interfaces such as ibp115s0f0, which may legitimately remain down. Each entry is a Go regular expression matched against the interface name; invalid regexps fail fast at startup.

By default the following interfaces are excluded: kernel tunnel fallback devices (gre*, gretap*, erspan*, ip6gre*, ip6gretap*, tunl*, sit*, ip6tnl*) and InfiniBand / IPoIB interfaces (ib*, ibp*). Setting excludedInterfaceNameRegexps overrides this default; set it to an empty list to disable exclusions entirely.

nodeAgent:
  monitors:
    networking:
      excludedInterfaceNameRegexps:
        - "^ibp[0-9]+s[0-9]+f[0-9]+$"
        - "^ib[0-9]+$"

The NVIDIA monitor supports dcgmPowerThresholdWatts to configure the DCGM power policy warning threshold for GPUs whose normal board power exceeds the library default. This setting controls when DCGM reports a power excursion and does not cap GPU power, so choose a threshold appropriate for every GPU on the node. When omitted, the existing behavior is unchanged: an existing policy is preserved or go-dcgm supplies its default threshold. Omitting the setting does not reset a threshold previously stored in the running hostengine. For example:

nodeAgent:
  monitors:
    nvidia:
      dcgmPowerThresholdWatts: 1000

The address of the ipamd introspection API can be overridden with the IPAMD_INTROSPECTION_ADDRESS environment variable (default localhost:61679). Set it when ipamd does not listen on localhost, for example when the VPC CNI is configured with INTROSPECTION_BIND_ADDRESS=169.254.1.1:61679:

nodeAgent:
  ipamdIntrospectionAddress: "169.254.1.1:61679"

Config File Format

The agent reads a YAML config file mounted at /etc/nma/config.yaml. Omitted monitors default to enabled.

monitors:
  kernel-monitor:
    enabled: true
  networking:
    enabled: false
  storage-monitor:
    enabled: true
  nvidia:
    enabled: true
  neuron:
    enabled: false
  runtime:
    enabled: true

Valid plugin names: kernel-monitor, networking, storage-monitor, nvidia, neuron, runtime.

When a monitor is disabled:

• Its health checks are not executed.
• The corresponding NodeCondition (e.g., NetworkingReady) is not set on the node, avoiding false-positive healthy status for unmonitored subsystems.

Building

# Build the binary
make build

# Run tests
make test

# Build container image
make docker-build

Running checks on a non-Linux host

Some packages (for example the NVIDIA DCGM monitor) use cgo and build only on Linux, so make test cannot run or vet them on macOS. To run the lint checks (gofmt + go vet) and unit tests on Linux from any host, run them in a container:

# Uses Docker by default; set CONTAINER_TOOL to use finch, podman, etc.
make test-in-container
make test-in-container CONTAINER_TOOL=finch

This builds a small test image (Go toolchain plus libsystemd-dev) once, tagged by the Go version in go.mod, and reuses it on subsequent runs. Force a rebuild with make test-image FORCE=1.

The container resolves Go modules using your host’s module settings (GOPROXY, GOSUMDB, GONOSUMDB, GOPRIVATE, GONOPROXY, GOFLAGS), taken from the environment or go env. If proxy.golang.org is unreachable on your network, set a proxy that works for you, for example:

make test-in-container CONTAINER_TOOL=finch GOPROXY=direct

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines on:

• Reporting bugs and feature requests
• Submitting pull requests
• Code of conduct
• Security issue notifications

Security

If you discover a potential security issue, please report it via the AWS vulnerability reporting page. Do not create a public GitHub issue for security vulnerabilities.

See CONTRIBUTING.md for more information.

License

This project is licensed under the Apache-2.0 License. See LICENSE for the full license text.