1 pulling image moby/buildkit:buildx-stable-1 3.0s done

1 ERROR: failed to inspect pulled image moby/buildkit:buildx-stable-1:

  Error response from daemon: 404 page not found

make: *** [Makefile:168: container] Error 1

Root cause: the composite action ran ‘eval $(minikube docker-env) && make container’, which points ‘docker buildx build –load’ at minikube’s internal containerd. That path has no pre-configured buildx builder, so buildx autobootstraps one by pulling ‘moby/buildkit:buildx-stable-1’ inside minikube — a path that has been consistently returning 404 on ‘docker inspect’ after the initial manifest fetch for the past week.

Fix:

Add docker/setup-buildx-action (same pin the release workflow uses) to explicitly provision a buildx builder against the host Docker daemon, with a driver image pinned by the action’s own release.
Drop ‘eval $(minikube docker-env)’ before ‘make container’. The image is built against the host daemon (using the buildx builder from step 1) and then explicitly pushed into minikube via ‘minikube image load’. This mirrors the digest-validation branch of the same composite action, which already uses ‘minikube image load’ rather than swapping the Docker socket.

VERSION extraction from versions.txt is straight from the Makefile so a bumped operator version doesn’t require a workflow edit.

Only affects the source-build path; the digest-validation path is untouched.

ci: Fix webhook readiness race in operator integ-test setup

The ‘Wait for operator ready’ step used:

kubectl wait –for=condition=Ready pod –all -n amazon-cloudwatch

‘kubectl wait –all’ succeeds immediately when zero pods match, which is the normal state right after ‘make deploy’ (the ReplicaSet has not yet created the operator pod). The workflow then proceeded to apply an AmazonCloudWatchAgent CR while the webhook pod was ~1s old, failing with:

failed calling webhook “mamazoncloudwatchagent.kb.io”: … connect: connection refused

Replace it with:

‘kubectl rollout status deployment/cloudwatch-controller-manager’ — tracks desired vs available replicas, no zero-object race.
Poll the cloudwatch-webhook-service Endpoints object until it has at least one address — a Ready pod does not guarantee the service endpoints are programmed, and CR admission goes through that service.

ci: Gate integ-test start on webhook actually serving admission

The previous gate (deployment rollout + Endpoints populated) still raced: the manager Deployment in config/manager/manager.yaml has no readiness probe, so the pod reports Ready the instant the container starts, rollout completes ~1s after apply, Endpoints get an address immediately — all before the webhook TLS server has bound its port. The first CR apply then fails with ‘connection refused’.

The binary’s /readyz is a plain healthz.Ping (main.go) and does not gate on GetWebhookServer().StartedChecker(), so no pod-level gate can close this race without a source change.

Instead, probe the webhook end-to-end: poll ‘kubectl apply –dry-run=server’ of a minimal AmazonCloudWatchAgent CR until admission succeeds. Server-side dry-run passes through the full mutating+validating webhook chain without persisting anything, making it an exact readiness signal for what the tests do next.

Amazon CloudWatch Agent Operator

The Amazon CloudWatch Agent Operator is software developed to manage the CloudWatch Agent on kubernetes.

Supported Languages:

Java
Python
.NET
NodeJS

This repo is based off of the OpenTelemetry Operator

Build and Deployment

Image can be built using make container
Deploy kubernetes objects to your cluster make deploy

Pre requisites

Have an existing kubernetes cluster, such as minikube

Install cert-manager on your cluster

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml

Getting started

Set a shortcut for kubectl for the operator namespace

kubectl config set-context --current --namespace=amazon-cloudwatch

Look at all resources created

kubectl get all

Look at the manager pod logs to ensure the manager is functioning and waiting for workers

kubectl logs amazon-cloudwatch-agent-operator-controller-manager-66f67f47f78

You should see logs that look similar to below

{"level":"info","ts":"2023-06-29T01:37:36Z","msg":"Starting workers","controller":"amazoncloudwatchagent","controllerGroup":"cloudwatch.aws.amazon.com","controllerKind":"AmazonCloudWatchAgent","worker count":1}

Create an AmazonCloudWatchAgent resource

kubectl apply -f - <<EOF
apiVersion: cloudwatch.aws.amazon.com/v1alpha1
kind: AmazonCloudWatchAgent
metadata:
  name: cloudwatch-agent
  namespace: amazon-cloudwatch
spec:
  mode: daemonset
  serviceAccount: cloudwatch-agent
  config: |
    {
        // insert cloudwatch agent config here
    }
  volumeMounts:
  - mountPath: /rootfs
    name: rootfs
    readOnly: true
  - mountPath: /var/run/docker.sock
    name: dockersock
    readOnly: true
  - mountPath: /run/containerd/containerd.sock
    name: containerdsock
  - mountPath: /var/lib/docker
    name: varlibdocker
    readOnly: true
  - mountPath: /sys
    name: sys
    readOnly: true
  - mountPath: /dev/disk
    name: devdisk
    readOnly: true
  volumes:
  - name: rootfs
    hostPath:
      path: /
  - hostPath:
      path: /var/run/docker.sock
    name: dockersock
  - hostPath:
      path: /var/lib/docker
    name: varlibdocker
  - hostPath:
      path: /run/containerd/containerd.sock
    name: containerdsock
  - hostPath:
      path: /sys
    name: sys
  - hostPath:
      path: /dev/disk/
    name: devdisk
  env:
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    - name: HOST_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: K8S_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
EOF

Create Instrumentation resource

kubectl apply -f - <<EOF
apiVersion: cloudwatch.aws.amazon.com/v1alpha1
kind: Instrumentation
metadata:
  name: java-instrumentation
  namespace: default # use a namespace with pods you'd like to inject
spec:
  exporter:
    endpoint: http://cloudwatch-agent.amazon-cloudwatch:4316/v1/metrics
  propagators:
    - tracecontext
    - baggage
    - b3
    - xray
  java:
    env:
      - name: OTEL_METRICS_EXPORTER
        value: "none"
      - name: OTEL_LOGS_EXPORTER
        value: "none"
      - name: OTEL_AWS_APPLICATION_SIGNALS_ENABLED
        value: "true"
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: "http/protobuf"
      - name: OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT
        value: "http://cloudwatch-agent.amazon-cloudwatch:4316/v1/metrics"
EOF

Helpful tools

This package uses kubebuilder markers to generate kubernetes configs. Run make manifests to create crds and roles in config/crd and config/rbac
Generate deepcopy.go by running make generate

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.