"we don't think you'll fit in and follow the rules and you have to constantly prove that you can"

I speculate that if you were a permanent minority instead of a visiting inconvenience, then that 'nice' racism you describe would metastasize into the type of racism you see in the USA. It's more friction from time and exposure added on. And, you know, slavery.