Using OSM’s IngressBackend or injecting NGINX directly with the OSM sidecar?

OSM has a specific CRD (Custom Resource Definition) to complement ingress rules defined for the ingress controller. While the official documentation recommends not injecting NGINX, it turns out that the IngressBackend resource has one significant shortcoming as of May 2023: it doesn’t play well with pods that have multiple application containers exposed to the ingress controller. Although the best practice is to have a single application container per pod, you can’t respect this in some cases. In such a situation, Envoy (driven by OSM) keeps talking to only one of the application containers. To take a concrete example, consider the following setup:

Figure 1 – Multiple containers belonging to the same OSM-injected pod exposed through NGINX

In the above diagram, the red path is the one that doesn’t work, even when the initial call is intended to port 8080. This leads to confusion since Envoy (driven by OSM) silently forwards to 8082, corresponding to a different API. This leads to a situation where the caller might want to call:

• hostname:8080/api/api1controller/…

but Envoy will do:

• hostname:8082/api/api1controller/…

which usually results in a 404 call since API2 behind 8082 does not usually have a similar endpoint to API1. Things could be worse if both APIs share the same endpoint names, as troubleshooting is tough. Keeping the same setup but removing OSM from the mix is charming.

So, as of May 2023, if you are in such a situation, the best thing to do is inject the NGINX controller instead of  IngressBackend. That is how Linkerd deals with ingress as well. Note that we worked with NGINX F5 free edition, not the community one. Important to note that injection only works fine when using the VirtualServer, VirtualServerRoute, and Policy CRDs exclusively. It doesn’t work with the regular ingress rules.