This reminds me that in say, a *BSD ports tree, you end up pulling tarballs from the internet, extracting them and running make. (Granted there can be a hash on them so that's some verification)

But an exploit using that would likely sooner just write something malicious in the Makefile if it wants to compromise the build machine. Targeting the compiler for such a goal seems like it would be harder.

There’s the mandatory hash, but generally speaking: that’s how you build software on Unix/Linux; there’s no way around it. You can do what Poudriere automatically does on FreeBSD, which is doing the whole thing in a dedicated jail.