Quite a few things could be happening here. You can find most of the answers in your shell's manual, but those are usually incredibly long and oblique, so...
Chances are your problem boils down to one of a few things.
If your profile or bashrc have expensive things, consider trimming them back.
If your profile or bashrc use a reverse DNS lookup (to set the prompt or something), fix DNS or use hostname instead.
Shells open a lot of files, among other things, while initialising. If system load is high, it will often show up here.
If the banner is pre-authentication, it might also actually be authentication (pam, LDAP, etc.) that is slow.
It might be none of these things, though. A surprising amount of stuff happens right before displaying the prompt!