Azure Kubernetes Services – A typical error message when pulling images for a very atypical reason

The HTTP return code 401 (UnAuthorized) is extremely popular. Every ITer has encountered that at least once. If you happen to encounter that exception in the context of AKS trying to pull an image, the most obvious reasons could be:

• You did not grant proper permissions to the target Azure Container Registry (assuming you use ACR).
• At some point in time, it was ok, hence the reason you have existing containers running the same image successfully, but the configuration has changed in the meantime
• You might have existing running containers, but you are now dealing with a deployment that has an explicit imagePullPolicy set to Always, thus forcing AKS to pull the image from the registry again, revealing the issue.

If you have reviewed all the above but still encounter 401, this could be for a quite different reason, and let us say that the 401-return code is very misleading in this situation. I faced this issue with a colleague at work and we lost quite some time figuring out what the problem was…It turned out that this was simply due to a corrupted image. Not sure how it ended up in that state, but the way we found it out, is thanks to a local docker pull of the very same image. Docker returned a much more sensible exception stating something about an unknown blob, which set us on the right track. We could not even delete that image from the ACR anymore, but at least we knew what the problem was. So, we pulled the image from the initial registry again, re-tagged it, and pushed it onto ACR. Then, we opened a case to MS support to get rid of the messy image.

Hopefully, this tip will spare some people’s time.