Answer by sCiphre for Why does it take tens of seconds to get a shell prompt?

It's probably either waiting for DNS or trying to authenticate via LDAP or such.

Try to add UseDNS no to /etc/ssh/sshd_config

If it also does it on local logons, check if any LDAP servers or DNS servers you have configured are slow or unresponsive.