The downside is that nothing is installed by default. You have basically the bare minimum to boot and run.
Need Apache? you will need to install it. Need to send email? You will need to set that up. Need python? Ditto. And hope you like working at the command line because you have no X. Etc.
It's a nice way to get a system that does just what you need it to do and nothing more, but "minimal" installs are rarely useful all by themselves; they are a starting point.
IIRC, Ubuntu Server does not install X11 and it does not install Apache (unless you chose "Web Server" role or tick "Apache" checkbox in "Custom" role at install time) or any MTA or MDA.
Need Apache? you will need to install it. Need to send email? You will need to set that up. Need python? Ditto. And hope you like working at the command line because you have no X. Etc.
It's a nice way to get a system that does just what you need it to do and nothing more, but "minimal" installs are rarely useful all by themselves; they are a starting point.