You could try SSH certificates using something like https://smallstep.com/sso-ssh/ - essentially you delegate validation of your public key to a IDP, which your servers are configured to trust.
The other approach would be something like ansible or puppet to deploy trusted keys to all servers
Hm... these are both interesting but might be a bit overkill IMO.
I don't think I'd need a CA and intermediary step if all SSHd needs to do is check if a key is a currently approved key for this particular service or not; and I last looked at chef/puppet many years ago, and it was way too much orchestration work that we no longer need w/ Docker containers and smaller footprint host OSes.
I would switch to certificate based SSH authentication.
All the server keys gets signed by your CA, all clients also gets signed by your CA. Everyone implicitly trust eachother though the CA and it's as safe as regular SSH keys.
You can also sign short lived client keys if you want to make revocations easier, the servers don't care because now all it cares is that it's a valid cert issues by the CA, which can be done entirely offline!
HashiCorp Vault can also help managing the above, but it's also pretty easy to do manually.
I do this, use the small step ca/Cli to manage the lot. It's amazing
It's such an underrated feature. It baffles me how people immediately turn to overly complicated solutions solving a problem they don't really have to solve, just because everyone assumes the only way is the default commonly known way. Like OP, people immediately jump to the conclusion you need extra software to manage the keys, rather than using another authentication method natively supported, and keep filling their known_hosts file with junk, making the whole validation process useless because everyone just accepts whatever key the host presents.
It's amazing how simple it is. Developer needs temporary access to debug a web server? Sure, here's your 2h valid cert to log in as the web user on the server, don't even need to actually log into the server to put their key in and then remove it. I mint a cert and it's ready to go on whichever users and servers I specified in the cert. Can't even gain persistence because regular authorized_keys is disabled and we have limited session durations.
I regularly leave people baffled at work because I come up with a clever and built-in super simple solution to something they expected to just slap more scripts and software to work around the only way they know to use the software. Read your manpages in full folks, it'll save you so much work. Know what your software is capable of.
Are you initiating SSH connections from all these hosts?
If you just need to SSH to these hosts, use a single key and copy the public key only to the hosts you need to connect to. If you don't want to copy the pubkeys to target hosts, use LDAP + SSSD or certificates.
Then, if you do need to initiate connections from these hosts and use an SSH agent you can forward your agent and SSH to another host
client> ssh -A host1
host1> ssh host2
host2>
client> ssh -A host1
host1> ssh -A host2
host2> ssh -A host3
host3>
Have an alias so trusted hosts can bounce through my authorization host and end up on a tmux session on the targetted host. It has logging and such but mostly it's for simplicity.
If I plan to use that connection a lot there's a script to cat my priv key through the relay.
Have an scp alias too, but that gets more complicated.
For more sensitive systems I have 2fa from gauth set up, works great.
This is a common pattern, typically called a "jump host" or "bastion host".
a script to cat my priv key through the relay
When it comes to security, I typically recommend against rolling your own. SSH already has agent forwarding option to do this securely and the -J option to accomplish the same without even needing to forward the key. The agent can seem complex at first, it's actually pretty simple and worth learning.
Feel free to message me if you have more questions, I've got lots of experience w/ SSH.
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
Any issues on the community? Report it using the report flag.
Questions? DM the mods!