Troubleshooting Kubernetes Ingress Controller

In fact troubleshooting Kubernetes ingress controller is easy and it could save you a lot of time. You just need to know when it is a good idea to do so. Here is an example when you may need it: a request produces an error, but no error logs are reported by the pod responsible for it.

When I was building a Drupal site, I noticed that adding a new field on the page produced 502 Bad Gateway Error. I thought the problem was in the field type that caused a fatal PHP error, but usually such errors are reported, however, I was not able to find anything in the logs of pod running Apache and PHP. After going back and forth, I decided to check logs of the Nginx ingress controller.

Typically there is a Nginx pod responsible for ingress controller running in each Kubernetes node in the cluster. So the first step would be getting all such pods.

Running following command:

kubectl get pods --all-namespaces | grep ingress-nginx

Will output something like:

ingress-nginx    default-http-backend-49cd995543-kaff4
ingress-nginx nginx-ingress-controller-bqb4a
ingress-nginx nginx-ingress-controller-gqrb3
ingress-nginx nginx-ingress-controller-agsrd
ingress-nginx nginx-ingress-controller-24w2g
ingress-nginx nginx-ingress-controller-q7pzs

The first column is a namespace, the second one is a pod name.

For each pod (as you are not sure which one is used), you can get logs with a following command:

kubectl logs -n ingress-nginx <nginx-pod-name> | grep <url>

Running:

kubectl logs -n ingress-nginx nginx-ingress-controller-bqb4a | grep "node/add/person"

Outputs this:

2020/12/02 14:18:53 [error] 605#605: *436444 upstream sent too big header while reading response header from upstream, client: 172.18.70.5, server: example.org, request: "GET /node/add/person HTTP/1.1", upstream: "http://10.42.64.3:80/node/add/person", host: "example.org", referrer: "https://example.org/node/add"

As we can see the actual error is upstream sent too big header while reading response header from upstream. This is caused because Drupal is setting quite a long Cache-Tags header.

The solution was to update Ingress Configuration to something like this.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
generation: 2
name: web-ingress
labels:
app: web
annotations:
nginx.ingress.kubernetes.io/proxy-buffer-size: 10k
spec:
rules:
- host: example.org
http:
paths:
- backend:
serviceName: web-service
servicePort: http

Depending on the issue, you can tweak Nginx configuration, see https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/ to have an idea what you can set.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store