Short-term tenants generally pay substantially more on a per night basis than real roommates (long-term tenants). Forcing only long-term tenants would thus presumably cause rents to move towards what the long-term tenant market can handle rather than the aggregate nightly rate achievable by accommodating lots of short-term tenants.
Naturally the reality is vastly more complicated but I believe that's the basic reasoning behind the "short term rentals drive prices up" line of thought.
Not only that, but some owners may decide to go full short-term leases to increase profits. That reduces the market for long term leases, hurting the housing market.
Now this is definitely the case for cities like Paris, were the law exists but is not enforced. I don't know of it's true for San Francisco where the price is already high from high paid locals.
Short term tenants are also exempt from the laws that protect SF nightmare tenants and having to pay $40k to move someone out. I know someone who has a legal in law unit in SF who basically decided not to ever rent it out because of those stories.
Naturally the reality is vastly more complicated but I believe that's the basic reasoning behind the "short term rentals drive prices up" line of thought.