When an SCP outranks your IAM grant

I was standing up a small serverless app — DynamoDB, S3, a few Lambdas behind an HTTP API, Cognito for auth — and doing the deploy through a locked-down CodeBuild role that's the only principal allowed to touch Lambda code in that account. Standard SAM: sam build, sam deploy. It failed:

User is not authorized to perform: cloudformation:CreateChangeSet on resource:
arn:aws:cloudformation:us-east-1:aws:transform/Serverless-2016-10-09

Fine — the role's missing a permission. I added it:

{
  "Effect": "Allow",
  "Action": "cloudformation:CreateChangeSet",
  "Resource": "arn:aws:cloudformation:*:aws:transform/Serverless-2016-10-09"
}

Same error. I broadened the resource to transform/*. Same error. I checked for a permissions boundary on the role — none. I re-read the message three times to make sure I wasn't misreading the ARN. It was right. The grant was right. It still said no.

The thing I kept forgetting

That arn:aws:cloudformation:*:aws:transform/Serverless-2016-10-09 resource is the SAM macro — the transform that expands AWS::Serverless::Function into raw CloudFormation. CreateChangeSet has to call it, so the caller needs permission on the transform itself, not just on the stack.

But the deny wasn't coming from a missing Allow. It was coming from an explicit Deny in a Service Control Policy at the Organizations level. And SCPs sit above IAM in the evaluation order: an SCP Deny is a hard ceiling. You can grant a principal cloudformation:* on * and it changes nothing — the request never gets to the IAM allow/deny stage because the org already said no.

The tell was the shape of it. A normal missing-permission looks the same in the error string, so I burned two iterations treating it like one. The signal I should have read sooner: the grant is unambiguous and present, and the answer didn't change. When adding the exact permission the error asks for has zero effect, stop adding permissions. Something above IAM is talking.

The fix: stop asking the macro

I didn't need SAM. SAM's only job here was to expand sugar into the same resources I could create directly. So I threw out the template and wrote an idempotent boto3 deploy script that builds each resource with the granular service permissions the role did have — dynamodb:CreateTable, s3:CreateBucket, lambda:CreateFunction, apigatewayv2:*, cognito-idp:* — none of which touch a CloudFormation transform.

The shape that made it painless to re-run on every push:

def ensure_table(name):
    try:
        ddb.describe_table(TableName=name)
        return                      # already there, nothing to do
    except ddb.exceptions.ResourceNotFoundException:
        pass
    ddb.create_table(...)
    ddb.get_waiter("table_exists").wait(TableName=name)

def ensure_function(name, handler, role_arn, code, env):
    try:
        lam.get_function(FunctionName=name)
        lam.update_function_code(FunctionName=name, ZipFile=code)
        # ...update_function_configuration
    except lam.exceptions.ResourceNotFoundException:
        lam.create_function(FunctionName=name, Code={"ZipFile": code}, ...)

Every step is "describe, and create-or-update from what you find." Run it once, it builds the stack. Run it a hundred times, it converges to the same state. The one gotcha worth keeping: a brand-new IAM role isn't immediately assumable, so create_function can throw InvalidParameterValue: cannot be assumed for a few seconds after the role is made — wrap it in a short retry loop.

I lost the drift-detection and clean teardown that CloudFormation gives you for free. For a small app deployed from one pipeline, that's a fine trade. I got a deploy that runs in the account exactly as it is, instead of one that fights a policy I don't control and can't see from inside IAM.

The lesson, minus the app

Two strikes on one hypothesis means the hypothesis is wrong. Re-applying the same fix — re-adding the permission the error names — after it already failed once is the trap. If the exact grant the error asks for doesn't move the needle, the deny is coming from somewhere IAM can't override: an SCP, a permissions boundary, a resource policy. Go find that layer before you edit one more policy document.