It seems that just using the ssh-agent-plugin solves my problem. I thought I tried that in the past.