Debugging Apptainer (and other Go code) with Delve and VScode

It is possible to configure your environment to start Apptainer (with arbitrary commands and options/arguments) in a Delve debugging session and then attach to the running server with VSCode.  This gives you the ability to set break points, step in/out/over lines of code and functions, interrogate variables during runtime, etc. 

But it’s a little tricky to set it up and I always forget how.  So I’m recording the setup and the run steps for myself and others.

This is tested on Rocky 9 but should work (more or less) on other distributions, too. 

Setup

1. Install VSCode and the Go extension. 

2. git clone your development Apptainer repo. 

Make a directory and git clone your Apptainer development repo there.  Here we assume it will go in /home/$USER/Repos/apptainer.  Edit as necessary.

1   mkdir ~/Repos

2   cd !$

3   git clone https://github.com/apptainer/apptainer.git

4   cd apptainer

5   git checkout vX.X.X #optional

3. Install Apptainer.

Apptainer has to be installed on your system for debugging to work.  If you already have it installed, just make sure you have the exact same version of the repo checked out.  If you are not sure, it might be best to just install a development version of Apptainer in a different location and set your PATH accordingly.  (Hint: Use the --prefix flag in ./mconfig).

Note: it might be best to use the version of Go in your distribution repos if it is new enough to support Apptainer.  This might avoid issues with Go/Delve mismatch later. 

4. Install Delve and dependencies.

1   sudo dnf config-manager --enable crb

2   sudo yum install delve gpgme-devel

Note: you might want to install delve via sudo go get -u github.com/go-delve/delve/cmd/dlv if you installed go from the precompiled binary (tarball).  See this issue for an example. 

5. Create a launch.json

Of course, there are a few different ways to do this.  You can do it from within VSCode (the debug panel gear widget in the upper right).  But this should work, too.

mkdir ~/Repos/apptainer/.vscode

cd !$

cat>launch.json<<EOF

{

    // Use IntelliSense to learn about possible attributes.

    // Hover to view descriptions of existing attributes.

    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387

    "version": "0.2.0",

    "configurations": [

        {

            "name": "Attach",

            "type": "go",

            "request": "attach",

            "mode": "remote",

            "port": 2345,

            "host": "127.0.0.1",

            "showLog": false

        }

    ]

}

EOF

6. Create an alias for the debug command (optional).

I can never remember this command so I just create an alias and save it in my .bashrc.  Of course, you can just type it out every time or just create the alias during this single session or whatever. 

Whatever you do, this command is going to take the place of the apptainer command when you want to debug stuff.  

alias db-app='cd /home/${USER}/Repos/apptainer/cmd/apptainer && dlv debug --headless --listen=:2345 --api-version=2 --log --'

Execution

1. Start the debugging session in the Delve server. In a terminal window, execute something like the following.  (The exact syntax will depend on the alias you set [or didn’t set] above). 

db_app run docker://godlovedc/lolcow

You will see a bunch of info messages scroll by ending with something like concrete subprogram without address range at and then the terminal will appear to hang.  The server is waiting for you to attach. 

2. Open ~/Repos/apptainer/cmd/apptainer/cli.go (or wherever you saved it). I like to open the apptainer directory itself in VSCode and then open the file separately.  But whatever you do, this is the actual executable recognized by Delve.

3. Attach your VSCode to the running server.

Set a breakpoint somewhere (by clicking to the left of the code line numbers until a red dot appears).  Then in the debug panel on the left, either click the green triangle next to “Attach“ or just press F5.  (Function keys are your friend in debug mode.)

4. “Go“ debug your code. “Step“ to it!  

Gotchas

• Is gpgme-devel installed?  (It needs to be.)  It’s in the PowerTools repo.

• When you try to attach are you seeing info messages appear in the terminal with the Delve server?  If not double check your launch.json

• Does Delve complain that it is too old for the Go version?  Maybe you need to install it with go get instead.  Check here.

• Is the delve server hung?  There may be a better way to do this, but I find I sometimes need to find and kill the PID.  In a new terminal…

ps -ef | grep dlv

kill <PID>