If you want to play a Minecraft server with a mod, do you need to download the mod first?

Yes, unless the server has client side mod(s) like mini maps, if it adds any new blocks/items or crafting recipes then you will need to install the mod(s) as well.

You not only have to have the same mods, you have to have the same versions of the same mods.

Maybe