Late last week I needed to test some Kubernetes functionality, so I thought I’d spin up a test cluster really quick using Cluster API (CAPI). As often happens with fast-moving projects like Kubernetes and CAPI, my existing CAPI environment had gotten a little out of date. So I updated my environment, and along the way picked up an important change in the default behavior of the clusterawsadm tool used by the Cluster API Provider for AWS (CAPA). In this post, I’ll share more information on this change in default behavior and the impacts of that change.
The clusterawsadm tool is part of CAPA and is used to help manage AWS-specific aspects, particularly around credentials and IAM (Identity and Access Management). As outlined in this doc, users use clusterawsadm to create a CloudFormation stack that prepares an AWS account for use with CAPA. This stack contains roles and policies that enable CAPA to function as expected.
Here’s the change in default behavior:
- In
clusterawsadm0.5.4 and earlier, usingclusterawsadmto create or update the CloudFormation stack would also create a bootstrap IAM user and group by default. - In
clusterawsadm0.5.5 and later, creating or updating the associated CloudFormation stack does not create a bootstrap IAM user or group.
This change in default behavior is briefly documented in the 0.5.5 release here. As mentioned in the release, the default behavior can be changed with a configuration file (API reference is available here).
In and of itself, this change in default behavior isn’t significant. What is significant is what happens if you use clusterawsadm 0.5.4 or earlier to create the necessary CAPA stack, and then use clusterawsadm 0.5.5 or later to update this stack. In such cases, if you haven’t taken steps to change the default behavior then the bootstrap IAM user and group are removed. When this happens, you’ll start to see error messages like this (or similar):
The user with name bootstrapper.cluster-api-provider-aws.sigs.k8s.io cannot be found
If your CAPI management cluster is using those credentials to interact with AWS, the CAPA controllers on that management cluster are now broken. You’ll have to update the CAPA controllers to use a new set of credentials (see this blog post for information on that process) before any CAPI-related operations will succeed.
One of the CAPA contributors (thanks, Naadir!) did point out that it is still possible to use the pre-0.5.5 clusterawsadm alpha commands in the 0.5.5 release. The CLI help text has been completely removed, but the command to run is clusterawsadm alpha bootstrap generate-cloudformation <aws-account-id> (this generates the CloudFormation template only; use clusterawsadm alpha bootstrap create-stack to actually create the stack). This command works with both the 0.5.4 and 0.5.5 releases of clusterawsadm, although the latter will generate a deprecation warning. However, the CloudFormation template generated by clusterawsadm 0.5.5 is not identical to the template generated by the 0.5.4 release; it lacks a name for the bootstrap IAM group. I have not tested what impact this has on existing CAPA stacks.
To get identical output (at least, with regard to the bootstrap user and group) between the two releases of clusterawsadm, you must generate a configuration file and make sure this section is present in the configuration file:
spec: bootstrapUser: enable: true userName: bootstrapper.cluster-api-provider-aws.sigs.k8s.io groupName: bootstrapper.cluster-api-provider-aws.sigs.k8s.io
Then specify the configuration file when running clusterawsadm:
clusterawsadm bootstrap iam create-cloudformation-stack --config config.yaml
Based on my testing, this should generate a CloudFormation stack that, with regard to the bootstrap IAM user and group, is identical to stacks created with clusterawsadm 0.5.4 and earlier. Thus, if you have existing CAPA environments prepared with clusterawsadm 0.5.4 and earlier, then—at least with regard to the bootstrap IAM user and group—it is safe to update these environments with clusterawsadm 0.5.5.
If anyone has questions, feel free to find me on the K8s Slack or hit me on Twitter. I’ll do my best to help.
