TIL: you can add the passwords of your ssh-keys to your Apple Keychain. And you can load all your keys into the ssh-agent that have saved passwords in the Keychain. With a small function (I use fish), you can load them on opening the first time your shell. But that part should be easily adaptable to any shell.

You need to use the ssh Apple provides. which ssh should result in /usr/bin/ssh

Do this once for all your ssh-keys:

ssh-add --apple-use-keychain $path/to/your/ssh-key

and then you do in the future once

ssh-add --apple-load-keychain

and all your ssh-keys will be loaded without having you entering your password.

I created then a small function in ~/.config/fish/functions - ssh_add_keys.fish:

function ssh_add_keys
  ssh-add -l > /dev/null || ssh-add -q --apple-load-keychain
end

After loading your ssh-agent (there are several fish-plugins like this one: fish-ssh-agent), you add ssh_add_keys.

This looks in my config.fish like this:

(...)

fish_ssh_agent
ssh_add_keys

(...)

When you open your terminal, your keys will be automagically added from the Apple Keychain; and this only when there no keys added to the ssh-agent yet.

Sources: