> in fact if people remember that secret then it's not a "second factor" it's just another part of their password.

This is more generous than it should be. Your TOTP secret is just another part of your password regardless of whether you personally remember it or not; what matters is that, if I would like to be you, I only need to know the secret.