Debugging the azure rm provider with vscode

Recently I’ve been learning Go and applying it to technologies that I work on everyday, like Terraform. For instance, check out my Terrahash CLI tool for validating modules. It’s neat!

I’ve also taken an interest in dipping my toe into the world of Terraform providers, and what better provider to start with than the azurerm provider I use for sooooooo many demos?

Of course, working on provider issues means debugging the provider itself and I had to learn how to set up debugging for VSCode. I thought I would document the process so you can see what I set up and possibly debug some providers yourself.

Shoutout to Drew Mullen and his excellent post on debugging the aws provider with VSCode. The azurerm provider is a little different, but the process is very similar.

Prerequisites

Before you get started with provider debugging on VSCode, you’ll need a few things in place:

• VSCode (I feel like that’s obvious, but 🤷)
• Go (Also kind of a given)
• VSCode Go extension (this includes delve which is necessary for debugging)
• Forked instance of the azurerm provider cloned to the correct directory
  • The correct directory being $GOPATH/src/github.com/hashicorp/terraform-provider-azurerm

Even though I run Windows, I prefer to do my Go development in WSLv2. In my experience, most Go tooling is intended for Mac and Linux with Windows as an afterthought. My examples will be using WSLv2, so fair warning on that.

How It Works

Since you’re debugging the provider and not the Terraform binary itself, you need to let Terraform know it should send provider requests to a listener on the debugger and not to the usual provider plugin. Terraform uses gRPC to communicate with providers, so it’s a simple matter of launching the provider in debugging mode and delve will handle setting up a gRPC listener on a UNIX socket.

Once the debugger finishes launching, it will produce a value for you called TF_REATTACH_PROVIDERS, which you can set as an environment variable for simplicity’s sake. Terraform looks for that environment variable and uses it to redirect provider plugin requests to the debugger.

After that, you can simply run Terraform commands like normal and if that command calls the provider and hits breakpoints in your code, it will pause and wait for you to resume.

The official Terraform docs for debugging providers uses a lot of fancy terminology that was mostly foreign to me. So I tried to simplify it down for someone who is not super familiar with Go or debugging, e.g. me.

Setting Up Debugging

VSCode gets its debugging settings from a file stored inside the .vscode directory in your repository. So first we need to create that directory and then a couple files.

From the root of the azurerm repository, run the following commands:

mkdir .vscode
touch .vscode/launch.json
touch .vscode/private.env

The launch.json file should contain the following:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug AzureRM Provider",
            "type": "go",
            "request": "launch",
            "mode": "debug",
            // this assumes your workspace is the root of the repo
            "program": "${workspaceFolder}",
            "env": {},
            "args": [
                "-debuggable",
            ],
            "showLog": true,
            "envFile": "${workspaceFolder}/.vscode/private.env"
        }
    ]
}

And the private.env should contain this:

TF_ACC=1
TF_LOG=INFO
GOFLAGS='-mod=readonly'

You can change the TF_LOG level if you want, but you’ll be getting all your info from the debugging process.

Using the Debugger

Once you have the files created, VSCode will show the name of the debug configuration in the debug tab:

Click on the green arrow to start the debugging process, and then bring up the DEBUG CONSOLE from the tabs on the integrated terminal.

You should see Delve starting up and listening on a local port:

Starting: /home/ned1313/go/bin/dlv dap --log=true --log-output=debugger --listen=127.0.0.1:44335 --log-dest=3 from /home/ned1313/go/src/github.com/hashicorp/terraform-provider-azurerm
DAP server listening at: 127.0.0.1:44335

It will take a while for the debugger to actually start, but once it does, you should see output like this:

TF_REATTACH_PROVIDERS='{"registry.terraform.io/hashicorp/azurerm":{"Protocol":"grpc","ProtocolVersion":5,"Pid":293831,"Test":true,"Addr":{"Network":"unix","String":"/tmp/plugin1338197622"}}}'

This is the environment variable you need to set to let Terraform know that it should redirect requests for the azurerm provider to the plugin listening at /tmp/plugin#######. The actual values will vary depending on the version of the provider and your system.

From the configuration you’d like to troubleshoot, first export the TF_REATTACH_PROVIDERS variable:

export TF_REATTACH_PROVIDERS='{"registry.terraform.io/hashicorp/azurerm":{"Protocol":"grpc","ProtocolVersion":5,"Pid":293831,"Test":true,"Addr":{"Network":"unix","String":"/tmp/plugin1338197622"}}}'

Then run your Terraform commands as usual. For instance, I have the following configuration:

terraform {
    required_providers {
      azurerm = {
        source = "hashicorp/azurerm"
        version = "~>3.0"
      }
    }
}

provider "azurerm" {
  features {
    
  }
}

resource "azurerm_resource_group" "example" {
  name = "debug-testing"
  location = "eastus"
}

I’ll run the following commands:

terraform init

$ terraform init

Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

terraform plan

$ terraform plan
azurerm_resource_group.example: Refreshing state... [id=/subscriptions/4d8e572a-3214-40e9-a26f-8f71ecd24e0d/resourceGroups/debug-test]

I added a breakpoint in the provider.go file to interrupt the process.

Once I continue through the breakpoint, the process continues and finishes.

Terraform will perform the following actions:

  # azurerm_resource_group.example will be created
  + resource "azurerm_resource_group" "example" {
      + id       = (known after apply)
      + location = "eastus"
      + name     = "debug-testing"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

And that’s it! You’re now debugging the azurerm provider. From here you can add breakpoints, inspect values, and look at the stack calls. I’ve found this invaluable for tracing down where an issue is occurring between Terraform core, the provider, and the Azure API.

Good luck and happy bug squashing!