A Go Module Testbed

I had recently lamented that due to Go’s strict module security policy it was unreasonably difficult to experiment and practice with modules. Modules can only be fetched from servers with valid TLS certificates, including both the module path and repository servers. Setting up a small, local experiment meant creating a certificate authority, generating and signing certificates, and installing these all in the right places. I’d much rather relax Go’s security policy for the experiment.

As a result of that complaint, I learned that the upcoming Go 1.14 has as a new feature: GOINSECURE. It’s like the old -insecure option, but safer due to being finer grained: a whitelist of exceptions. It’s exactly what I needed. Since then I’ve been using it to run small module experiments. It started as some scripts, but I eventually formalized it into its own little project.

https://github.com/skeeto/go-module-testbed [requires Go 1.14]

It’s first and foremost a shell script, and the Go source is only there as a server. The interface is like a Python virtual environment where “activating” the environment in a shell allows Go run from that shell to interact with the testbed servers. The script establishes the testbed environment and starts both servers in that environment. It optionally accepts a testbed directory as an argument, defaulting to the working directory as the testbed.

$ ./go-module-testbed

In addition to running the servers in the foreground, the script populates the testbed directory with an activate script, src/ containing module Git repositories, and www/ containing the static web server contents. These are initialized with a module named at v1.0.0. Why not localhost as the domain? The domain part of a module path must contain at least one dot, and IP addresses are acceptable.

The server logs requests to standard output so you can see each request Go makes to the server. This is has been an important part of learning what exactly Go is requesting from the web server hosting the module path.

There’s one giant caveat: Modules must be hosted on a privileged port. Normally that’s 443 (HTTPS), though in this case it’s 80 (HTTP). Since it’s a privileged port, you’ll need to do some system configuration. On Linux it’s easy enough just to temporarily forward the testbed port 8001 to port 80.

# iptables -t nat -I OUTPUT -p tcp -d \
           --dport 80 -j REDIRECT --to-ports 8001

Unfortunately this means, outside of doing something with namespace or containers, there can only be one testbed per host at at time. My goal is just to run small, local, temporary experiments, so this isn’t a big deal for me, but I wish it could be better.

Activating the environment

With the server running and the port forwarding configured, source the activate script from a shell:

$ source activate

This sets up an isolated, disposable GOPATH so that the testbed is completely isolated from your normal development. It also updates PATH, unconditionally enables modules (GO111MODULE=on), whitelists the testbed servers in GOINSECURE, and sets GOPRIVATE so that the testbed modules don’t leak anywhere outside the testbed environment.

The ensure that it’s all working, try installing the hello command from the example module:

$ go get
go: downloading v1.0.0
go: found in v1.0.0
$ demo
Example v1.0.0

Non-testbed modules are still accessible like normal, though all fetched and built artifacts are isolated in the testbed environment:

$ go get nullprogram.com/x/passphrase2pgp
$ go get golang.org/x/tools/cmd/goimports

So you can mix your experiments and practice with real modules.

Running experiments

From here you could practice creating a new minor version of the example module, and see how it appears to the module’s users.

$ sed -i s/v1.0.0/v1.1.0/ src/example/example.go 
$ git -C src/example/ commit -a -m 'Bump to v1.1.0'
[master 7a3cf82] Bump to v1.1.0
 1 file changed, 1 insertion(+), 1 deletion(-)
$ git -C src/example/ tag -a v1.1.0 -m v1.1.0
$ go get
go: downloading v1.1.0
go: found in v1.1.0
$ demo
Example v1.1.0

Or try more challenging: Release a v2.0.0, which requires changing the module path.

$ cd src/example/
$ go mod edit -module go.mod
$ sed -i s/v1.0.0/v2.0.0/ example.go 
$ git commit -a -m 'Bump to v2.0.0'
[master bf5c4cf] Bump to v2.0.0
 2 files changed, 2 insertions(+), 2 deletions(-)
$ git tag -a v2.0.0 -m v2.0.0
$ cd ../../www/example/
$ mkdir v2
$ sed 's#e git#e/v2 git#' index.html >v2/index.html
$ cd ../../
$ go get
go: downloading v2.0.0
go: downloading v1.0.0
go: found in v2.0.0
go: finding module for package
go: found in v1.0.0

I was able to figure this all out specifically because of my testbed. Adding a /v2 module path on the web server was not obvious, and it’s glossed over in the tutorials.

Nested modules

One of the under-documented corners of Go modules is nested modules. That is, repositories that contain more than one module. (Note: These are not called submodules since that would be confusing in the context of Git.) The Go module testbed is great place to try them out — and to learn why they should never be used. Even if I never plan to use them, I still want to understand them since I might need to debug them someday.

There are two tricky parts to nested modules: the version tag and the module path. Neither are documented as far as I’ve seen, so I had to figure them out from official examples.

$ mkdir src/example/nested
$ cd src/example/nested/
$ go mod init
go: creating new go.mod: module
$ echo package nested >nested.go
$ git add .
$ git commit -m 'Add a nested module'
[master c5b1a29] Add a nested module
 2 files changed, 4 insertions(+)
 create mode 100644 nested/go.mod
 create mode 100644 nested/nested.go
$ git tag -a nested/v1.2.3 -m v1.2.3
$ cd ../../../
$ mkdir www/example/nested
$ cp www/example/index.html www/example/nested/
$ go get
go: downloading v1.0.0
go: downloading v1.2.3
go: upgrade => v1.2.3

Module versions are derived from the Git tag, which is global to the repository. So how are nested modules versions indicated? They get namespaced tags, as shown above with nested/v1.2.3. If I didn’t create this tag, it would be as if I didn’t tag any version of that module.

The second unintuitive part is the web server’s response to ?go-get=1. At the nested module path, the response must indicate the containing module and where to get it. In other words, it’s the same response as the containing module, which is why I merely copied index.html. Returning a 404 for the module path is no good — another thing I’ve learned from the module testbed.

There are still many things I have yet to try or practice in my module testbed. It’s great that now when I have a niggling question about modules or go get behavior, I can get an answer within a minute or so without needing to dig through useless online search results.

Have a comment on this article? Start a discussion in my public inbox by sending an email to ~skeeto/public-inbox@lists.sr.ht [mailing list etiquette] , or see existing discussions.

null program

Chris Wellons

wellons@nullprogram.com (PGP)
~skeeto/public-inbox@lists.sr.ht (view)