Also put the temporary directory into the isolated environment.

This fixes each test run "leaking" an empty directory in /tmp, home of the
server socket. The server usually would reuse that on the next run, so
nothing wrong there, but of course this doesn't happen if we delete
everything else right after that.