KinD with MetalLB on macOS

I have been experimenting with the kubernetes Gateway API using kind in Docker Desktop for macOS. Theoretically I should be able to change implementations without having to modify my resources - almost. I started with this awesome Hands-On with the Kubernetes Gateway API: A 30-Minute Tutorial which uses Gloo Gateway v2 (beta). Under the hood, the edge proxying is done with Envoy, the same as Istio uses. I managed to work through the Gloo tutorial getting by with forwarding my curl requests through a kubectl port-forward but when I switched over to Istio I found a new problem.

The problem

Istio requires a load-balancer. Without one, all of your port-forwarded requests will return a 404 as the Istio control plane doesn’t know where to send your traffic.

❯ kubectl get gtw -A
istio-ingress   gateway   istio   <pending>    False        2d1h

Note the programmed=False condition which indicates the Gateway control plane has not been able to make a decision on its status:

❯ kubectl get gtw gateway -n istio-ingress -o yaml | yq .status.conditions\[1]
lastTransitionTime: "2024-06-02T06:09:41Z"
message: 'Assigned to service(s) gateway-istio.istio-ingress.svc.cluster.local:80,
  but failed to assign to all requested addresses: address pending for hostname
observedGeneration: 1
reason: AddressNotAssigned
status: "False"
type: Programmed

Any attempt to change the Service resource to a type: ClusterIP would just get replaced by istiod - it really wants to use a type: LoadBalancer!

NOTE: Docker Desktop Mac runs in a process-based VM using hyperkit and does not expose the VM’s network interfaces to the macOS host.

The setup

KinD allows to create a 2 node cluster with this configuration when you initialise the cluster:

kind: Cluster
- role: control-plane
- role: worker
❯ kind create cluster --config=cluster.yaml

❯ kind get clusters

❯ kubectl cluster-info --context kind-kind
Kubernetes control plane is running at
CoreDNS is running at

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

❯ kubectl get nodes
NAME                 STATUS   ROLES           AGE     VERSION
kind-control-plane   Ready    control-plane   2d21h   v1.30.0
kind-worker          Ready    <none>          2d21h   v1.30.0

The rest of the setup can be found in the Gloo tutorial mentioned above, but it’s basically a Gateway controlling a Service on port 8000 and routing a HTTPRoute to a httpbin pod on port 80.

The solution

I found a reddit post that referred to Docker Mac Net Connect

Connect directly to Docker-for-Mac containers via IP address.

It’s a pretty awesome idea - you create a utun virtual network interface on the macOS host which routes traffic over a lightweight wireguard virtual network to the kernel running in the VM.

As per their instructions, installation was simple:

# Install via Homebrew
$ brew install chipmk/tap/docker-mac-net-connect

# Run the service and register it to launch at boot
$ sudo brew services start chipmk/tap/docker-mac-net-connect

Once installed, we can see it is running the daemon process on the host:

❯ sudo brew services list
Name                   Status  User File
docker-mac-net-connect started root /Library/LaunchDaemons/homebrew.mxcl.docker-mac-net-connect.plist

❯ sudo brew services info docker-mac-net-connect
docker-mac-net-connect (homebrew.mxcl.docker-mac-net-connect)
Running: ✔
Loaded: ✔
Schedulable: ✘
User: root
PID: 3145

❯ ps aux | grep 3145
root              3145   0.0  0.1 35504240   9048   ??  Ss    3:39pm   0:00.09 /usr/local/opt/docker-mac-net-connect/bin/docker-mac-net-connect

Currently the project does not have a config file but it’s easy to see it created a new utun5 network interface and we can see the Docker networks are now routed to it automatically

❯ netstat -rnf inet | grep         UH                  utun5

❯ netstat -rnf inet | grep utun5         UH                  utun5
172.17             utun5              USc                 utun5
172.18             utun5              USc                 utun5

All that remains is to install MetalLB and configure it to act as a load balancer.

❯ kubectl apply -f
namespace/metallb-system created created created created created created created created
serviceaccount/controller created
serviceaccount/speaker created created created created created created created created created
configmap/metallb-excludel2 created
secret/metallb-webhook-cert created
service/metallb-webhook-service created
deployment.apps/controller created
daemonset.apps/speaker created created

We can get the worker node’s IP address with a simple docker command:

❯ docker exec -ti kind-worker ip addr show dev eth0 scope global
31: eth0@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc noqueue state UP group default
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet brd scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::2/64 scope global nodad
       valid_lft forever preferred_lft forever

Using that, we configure an IPAddressPool resource for MetalLB so it knows to start listening on that address:

kind: IPAddressPool
  name: kind-worker
  namespace: metallb-system
  autoAssign: true
  avoidBuggyIPs: false

Copy and paste it to kubectl on stdin to apply it:

❯ pbpaste | kubectl apply -f - created

Now we can see the Gateway resource has met the condition for a load-balancer

❯ kubectl get gtw -A
istio-ingress   gateway   istio   True         2d1h
❯ kubectl get gtw gateway -n istio-ingress -o yaml | yq .status.addresses,.status.conditions\[1]
- type: IPAddress
lastTransitionTime: "2024-06-04T06:13:13Z"
message: Resource programmed, assigned to service(s) gateway-istio.istio-ingress.svc.cluster.local:80
observedGeneration: 1
reason: Programmed
status: "True"
type: Programmed

Finally we can ping the worker node’s IP address and make HTTP requests to our pods via the Gateway without needing to port-forward anything.

❯ ping
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=63 time=3.833 ms
64 bytes from icmp_seq=1 ttl=63 time=0.885 ms
--- ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.885/2.359/3.833/1.474 ms

❯ curl -vs -H "Host:"
*   Trying
* Connected to ( port 80
GET /api/httpbin/get HTTP/1.1
User-Agent: curl/8.4.0
Accept: application/json, */*

HTTP/1.1 200 OK
server: istio-envoy
date: Tue, 04 Jun 2024 06:15:52 GMT
content-type: application/json
content-length: 1378
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 38

* Connection #0 to host left intact
    "args": {

    "headers": {
        "Accept": "application/json, */*",
        "Authorization": "Bearer my-api-key",
        "Host": "",
        "User-Agent": "curl/8.4.0",
        "X-Envoy-Attempt-Count": "1",
        "X-Envoy-Decorator-Operation": "httpbin.httpbin.svc.cluster.local:8000/*",
        "X-Envoy-Internal": "true",
        "X-Envoy-Original-Path": "/api/httpbin/get",
        "X-Envoy-Peer-Metadata": "ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKHQoMSU5TVEFOQ0VfSVBTEg0aCzEwLjI0NC4xLjE1ChkKDUlTVElPX1ZFUlNJT04SCBoGMS4yMi4wCvABCgZMQUJFTFMS5QEq4gEKMwomZ2F0ZXdheS5uZXR3b3JraW5nLms4cy5pby9nYXRld2F5LW5hbWUSCRoHZ2F0ZXdheQoiChVpc3Rpby5pby9nYXRld2F5LW5hbWUSCRoHZ2F0ZXdheQoyCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEg8aDWdhdGV3YXktaXN0aW8KLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0CiIKF3NpZGVjYXIuaXN0aW8uaW8vaW5qZWN0EgcaBWZhbHNlChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAonCgROQU1FEh8aHWdhdGV3YXktaXN0aW8tNTdmOGI0NGRmLXNzMm5iChwKCU5BTUVTUEFDRRIPGg1pc3Rpby1pbmdyZXNzClcKBU9XTkVSEk4aTGt1YmVybmV0ZXM6Ly9hcGlzL2FwcHMvdjEvbmFtZXNwYWNlcy9pc3Rpby1pbmdyZXNzL2RlcGxveW1lbnRzL2dhdGV3YXktaXN0aW8KIAoNV09SS0xPQURfTkFNRRIPGg1nYXRld2F5LWlzdGlv",
        "X-Envoy-Peer-Metadata-Id": "router~"
    "origin": "",
    "url": ""
comments powered by Disqus