Is it true that VAX allowed customers to extend the ISA themselves?

I think I learned about that many years ago, but I couldn't find anything about that recently when skimming the 11/780 user manual.

Yes. Search for “writable control store”.

https://en.wikipedia.org/wiki/Control_store#Writable_stores

Here’s an example usage:

http://hps.ece.utexas.edu/pub/gee_micro19.pdf

Was this about the KU780 option?

Absolutely correct, KU780 was the Writeable Control Store described here http://bitsavers.trailing-edge.com/pdf/dec/vax/handbook/VAX_...

Sophisticated customers hacking the instruction sets of their machines goes back pretty much to the beginning. The earliest I personally know of is Prof Jack Dennis hacking MIT's PDP-1 to support timesharing, sometime in 1961. Commercial machines like the Burroughs B1700 had a WCS that was designed so various compiled languages could be optimized - e.g. a FORTRAN instruction set, a COBOL instruction set, etc https://en.wikipedia.org/wiki/Burroughs_B1700 It was also in the IBM360s because they had to emulate the IBM1401 software (although I don't know if the capability was open to users to modify).

Today of course you have the various optional features of the RISC-V ecosystem --- easy to load up on an FPGA.

Perhaps we should remember that we are in the very very early days of Computers, and we should expect continued modification / experimentation.