How to register Elastic Agent outside of ECK

Table of Contents

ElasticSearch - This article is part of a series.

Part 5: This Article

Introduction
#

When using Elastic Cloud on Kubernetes (ECK), there may be times when you want to collect logs from physical devices or VMs outside of ECK.

In this case, since the agents inside ECK are not exposed externally, you need to set up agents outside of ECK and register them with Fleet. However, this process can be quite challenging and there are various issues.

Issues related to certificate problems when registering agents outside of ECK, with no documentation leading to confusion:
- https://github.com/elastic/cloud-on-k8s/issues/6853
Issues where agents become unhealthy when FleetServer is exposed via Ingress:
- https://discuss.elastic.co/t/eck-fleet-server-behind-ingress-elastic-agents-becoming-unhealthy/313234
Issues mentioning the causes of agents becoming unhealthy when FleetServer is exposed via Ingress:
- https://github.com/elastic/elastic-agent/issues/2762

This post will introduce these issues and their solutions.

1. The Problem
#

The reason why external Elastic Agents cannot register with the ECK fleet-server is fully documented in the following issue:

https://github.com/elastic/elastic-agent/issues/2762

2. Solutions
#

We will briefly touch on the causes and introduce the solutions.

1. Expose fleet-server externally via Ingress

To allow external agents to access the ECK fleet-server, you need to expose the fleet externally by some means. As mentioned in ECK in production environment, it is recommended to expose ECK with TLS, so using Ingress to expose ElasticSearch, Kibana, and Fleet is common.

Here is a sample of the Ingress configuration I used.

For ElasticSearch and Kibana, the official Helm Chart now supports Ingress. If you are using AWS or GCP, you may refer to this.

# SAMPLE (nginx ingress controller + cert-manager + let's encrypt)
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kibana-ingress
  namespace: elastic-stack
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - "*.k8s.hirohirolab.com"
    secretName: quickstart-example-tls # https://cert-manager.io/docs/tutorials/acme/nginx-ingress/
  rules:
  - host: elastic.k8s.hirohirolab.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: elasticsearch-es-http
            port:
              number: 9200
  - host: kibana.k8s.hirohirolab.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kibana-kb-http
            port:
              number: 5601
  - host: fleet.k8s.hirohirolab.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: fleet-server-agent-http
            port:
              number: 8220

If you choose a path other than the root directory (- path: /) for Ingress routing, it seems to result in 404 Not Found due to redirects. Github-issue: Document edge cases around Fleet/Agent setups with Ingress

2. Change Fleet communication to go through Ingress

Although Fleet is now exposed via Ingress, the default setting accesses it via Servicehttps://elasticsearch-es-http.elastic-stack.svc:9200, making it inaccessible to external agents.

Therefore, we need to change the fleet-server communication to go through Ingress.

before

# https://github.com/elastic/cloud-on-k8s/blob/c25ae520dbba3d83231b96fb9e10dc081546ff39/deploy/eck-stack/examples/agent/fleet-agents.yaml#L40
      xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.elastic-stack.svc:9200"]
      xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.elastic-stack.svc:8220"]

after

# THIS IS A SAMPLE. INPUT YOUR FLEET AND ELASTIC URL.
      xpack.fleet.agents.elasticsearch.hosts: ["https://elastic.k8s.hirohirolab.com"]
      xpack.fleet.agents.fleet_server.hosts: ["https://fleet.k8s.hirohirolab.com"]

This will switch Fleet communication to go through Ingress.

3. Certificate management for agents (problem)

This allows external agents to register, but now internal agents will encounter certificate errors as shown below.

The cause is that internal agents store a self-signed certificate generated by ECK in FLEET_CA.

root@v0-dev-03:~/project/kubernetes-ingress# kubectl exec -it eck-stack-with-fleet-eck-agent-agent-84thl -n elastic-stack -- /bin/bash
root@v0-k8s-03:/usr/share/elastic-agent# export | grep CA
declare -x FLEET_CA="/mnt/elastic-internal/fleetserver-association/elastic-stack/fleet-server/certs/ca.crt" # Insert By elastic-operator

Since we changed the fleet-server access to go through Ingress, it now uses the certificate registered with Ingress instead of the FLEET_CA certificate. As a result, certificate verification fails and an error occurs.

4 OverrideFLEET_CAandFLEET_URL

To resolve this, you can override FLEET_CA="" to use a general certificate (such as one from Let’s Encrypt or another external CA). Refer to the official documentation. Additionally, FLEET_URL needs to be changed to the Ingress URL.

root@v0-k8s-03:/usr/share/elastic-agent# export | grep FLEET_URL
declare -x FLEET_URL="https://fleet-server-agent-http.elastic-stack.svc:8220"

These environment variables are set by the elastic-operator, but according to the official documentation, they can be overridden.

In the official Helm chart, you add them under the daemonSet section.

# https://github.com/elastic/cloud-on-k8s/blob/c25ae520dbba3d83231b96fb9e10dc081546ff39/deploy/eck-stack/examples/agent/fleet-agents.yaml#L98
daemonSet:
    podTemplate:
    spec:
        serviceAccountName: elastic-agent
        hostNetwork: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        securityContext:
        runAsUser: 0
    # OVERRIDE ENV VARS
        containers:
        - name: agent
        env:
        - name: FLEET_CA
            value: ""
        - name: FLEET_URL
            value: "https://fleet.k8s.hirohirolab.com" # THIS IS A SAMPLE. INPUT YOUR FLEET URL.

5. Completion

Deploying with these settings will override the environment variables, and the status will become HEALTHY.

root@v0-dev-01:~/project/elasticsearch/cloud-on-k8s# kubectl exec -it eck-stack-with-fleet-eck-agent-agent-84thl  -n elastic-stack -- /bin/bash
root@v0-k8s-01:/usr/share/elastic-agent# export | grep -E "FLEET_CA|FLEET_URL"
declare -x FLEET_CA=""
declare -x FLEET_URL="https://fleet.k8s.hirohirolab.com"
root@v0-k8s-01:/usr/share/elastic-agent# elastic-agent status
┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   └─ status: (HEALTHY) Running

Conclusion
#

Conclusion With this setup, both external and internal agents can connect to the fleet-server via Ingress.

If TLS communication is not necessary or setting up Ingress is difficult, you can expose the fleet-server using LoadBalancer or NodePort and override with FLEET_INSECURE=1 to resolve the issue (untested).

Appendix
#

If you want to collect logs outside of ECK using Beats/Logstash instead of Elastic Agent, it’s simple. The official Helm chart already provides an example where Beats is exposed externally.

# https://github.com/elastic/cloud-on-k8s/blob/c25ae520dbba3d83231b96fb9e10dc081546ff39/deploy/eck-stack/examples/logstash/basic-eck.yaml#L105C1-L114C31
services:
  - name: beats
    service:
      spec:
        type: ClusterIP
        ports:
          - port: 5044
            name: "filebeat"
            protocol: TCP
            targetPort: 5044

Additionally, there is a guide on how to forward Syslog from Pure Storage (physical storage) to ECK. link

ElasticSearch - This article is part of a series.

Part 5: This Article

Introduction #

1. The Problem #

2. Solutions #

Conclusion #

Appendix #

Introduction
#

1. The Problem
#

2. Solutions
#

Conclusion
#

Appendix
#