I'm only a single sample, it's true. But I've been using headphones in one form or other pretty much daily for over 30 years and I've not yet noticed any deterioration (I have relatively poor hearing due to childhood disease, but that was before I started using headphones, and it's not got noticeably worse in the past 30+ years).

Also, noise cancelling headphones can allow you to listen to music at quieter than the office's ambient volume.