TIL: you can add the passwords of your ssh-keys to your Apple Keychain. And you can load all your keys into the ssh-agent that have saved passwords in the Keychain. With a small function (I use fish), you can load them on opening the first time your shell. But that part should be easily adaptable to any shell.
You need to use the ssh Apple provides.
which ssh should result in
Do this once for all your ssh-keys:
ssh-add --apple-use-keychain $path/to/your/ssh-key
and then you do in the future once
and all your ssh-keys will be loaded without having you entering your password.
I created then a small function in ~/.config/fish/functions - ssh_add_keys.fish:
function ssh_add_keys ssh-add -l > /dev/null || ssh-add -q --apple-load-keychain end
After loading your ssh-agent (there are several fish-plugins like this one: fish-ssh-agent), you add
This looks in my config.fish like this:
(...) fish_ssh_agent ssh_add_keys (...)
When you open your terminal, your keys will be automagically added from the Apple Keychain; and this only when there no keys added to the ssh-agent yet.