Solving Google GKE Autopilot permission denied to Cloud Storage, Datastore and other resources

Igor Rendulic -

Autopilot is by default using Workload Identity.

Workload identity is Google's way of connecting IAM accounts to Kubernetes service accounts so we don't need to import JSON credentials into Kubernetes secrets any more.

What we need to do is allow Pods to authenticate to Google Cloud APIs.

  1. Enable Google Cloud APIs on your project

By default the namespace of Autopilot cluster is named default and the Kubernetes Service Accounts (KSA_NAME) name is default.

2. Allow the Kubernetes service account to impersonate IAM service account by adding an IAM policy binding between two services (find an IAM service account you'd like the GKE Autopilot to bind to).

  • PROJECT_ID your google Project ID
  • GSA_NAME the name of your IAM service account
  • GSA_PROJECT the project ID of the Gogle cloud project of your IAM service account
  • KSA_NAME the name of the KSA (Kubernetes Service Account) which is default in GKE Autopilot
  • NAMESPACE which is default in the GKE Autopilot 
gcloud iam service-accounts add-iam-policy-binding GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com \    --role roles/iam.workloadIdentityUser \    --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

3. Annotate the Kubernetes service account with the email address of the IAM service account.

kubectl annotate serviceaccount KSA_NAME --namespace NAMESPACE \    iam.gke.io/gcp-service-account=GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com

And voila. It should magically work now without any 403 Permission denied problems anymore.

Resources