Follow @nielsk on Micro.blog.
It seems that just using the ssh-agent-plugin solves my problem. I thought I tried that in the past.