When have you EVER heard of an American-born white person being told that they should go back to the country from where they came? It just doesn’t happen.
But that phrase, and it’s variants, has a long history with people of color. I have personally heard white people say that to people of color, or more frequently, about people of color not to their face. I know you have, too.
To understand how people of color who have experienced this slur, you should check out this collage of letters written by ordinary American citizens who have personal experience with the phrase.
https://www.google.com/amp/s/www.lat...ml%3f_amp=true