Once you buy the argument that some segment of the citizenry should lose their rights, just because they are envied or resented, you are putting your own rights in jeopardy - quite aside from undermining any moral basis for respecting anybody's rights. You are opening the floodgates to arbitrary power. And once you open the floodgates, you can't tell the water where to go.
