As an epithet, Riff Raff, belongs to a category of insults focusing on behavioral characteristics (such as mannerisms, lifestyle) rather than overt racial characteristics (as with nigger, yid).
Other specific meanings pertain to music and films.
Music
Films