You can totally go to the UHS version of Hogwarts, it's identical to the one in Orlando. But IoA in Orlando has way more amazing rides in the park - The Hulk, fan-favorite ride Spiderman, Duelling Dragons, Revenge of the Mummy... Plus Orlando has Busch Gardens Tampa Bay with phenomenal rollercoasters and has Animal Kingdom with Expedition Everest where Disney literally built a fucking 200ft / 60m mountain for one amazing rollercoaster: If you mostly want to get the HP experience I understand why you'd scrap Orlando, but if you don't mind the super expensive park tickets and want the best experience, I'd put it back up your itinerary.
Ahah this is crazy. I will save money by skipping Orlando in order to go to Yosemite, the real mountains should be nicer :) I get the opportunity to go to the park with friends in LA while I would be mostly alone in Orlando, so that journey is going to be for another time in life !