I think it might be because of the little vibrations in the instruments sounds that we normally don't notice because when played at regular speeds they sound more like one whole note instead of a vibrating one. The low to medium-low notes of the piano for example, or the broken up sound of a shaker. At speed they sound fine, but when slowed down they wobble and scratch a bit more as our ears have more time to pick out individual bits of what's really going on. Also, the dissonant notes and runs within the chords pop out a ton more in the slowed version because they're given more time to their selves. Normally dissonant or jazzy notes don't sound too bad to us because they're short and eventually come to rest on a more pleasing chord. But in this, they're all stretched out so it comes off more harsh as you said, and yields a more dark, dissonant sound (which is why you see such dissonant notes drawn out in a lot of dark classical orchestra pieces). Just my thoughts. I could way totally be wrong.