I rather thing it's the way they live, too. Like, if a person dives into the Dark Arts, it corrupts them in a way, and so their bodies/wealth, eventually suffer.

Dorian Grey-like, you mean? That's fascinating; I'd love to read a story based on that. But - I could never make it my headcanon. It presupposes that "nature" has morals and punishes those who behave amorally. And well, nature has no morals. Morality is man-made, or part of a spiritual or religious belief. I'd love to know more, though, about why Voldemort looked the way he did, after his return.