A part of it is the reading level that is expected.
No, not that all science fiction difficult, but you’ve got 2 things going on in high-end science fiction:
1. Literary bent:
Some science fiction is very speculative, asking deep questions about the nature of man. Flowers for Algernon is a great example: that is often just labeled a classic or literary fiction, but it’s science fiction. They did a “miracle cure” for low IQ on a man, to see how it would work. That’s not even feasible right now. That means they had to “sell the science”, so I could suspend my disbelief, and thet did it through the ever-changing mind of the man being tested. Another one I talk about on here, calling it Feminist Anthem is Native Tongue. It’s set in a future where women’s right to vote and be considered legally competent was stripped from them, from the family of linguists who worked women as equal-yet-slaves, in translating alien languages, while making a language that men wouldn’t believe was real, so that women could talk without them monitoring what women said and did: you have to be a linguist to even think like this book, in order to write it. But it’s a book that questions the nature of language bolding women back from being able to express themselves succinctly. And it was doing things with language that slang is only getting to, now. The concept of “Brother from another mother” or “sister from another mister” was not succinctly expressed back when the book was written. It over-hyped men as finding women too emotional to reason, but it did deal with some real problems of language. There’s plenty that’s not succinctly covered in language. For example, if I could write djdjjjjsk, and this whole paragraph was covered by that gibberish, then I wouldn’t have to write all this out.
But it shows that there’s a strong bent in some science fiction of NOT being an escape from reality, but forcing your nose in the bad of reality like you did it and need to change.
2. Selling the future:
A light saber is just a magic sword. The force is just magic. But if you don’t get a little technical, you’re not selling the magic sword as a light saber. The requirements to make tech seem relatable and beyond us means borrowing from other tech and expanding on it. Hard Fantasy borrows from what is necessary in science fiction, to make a richer fantasy environment, but magic doesn’t inherently require that we “expand on the tech”.
So, that’s the part where it is foundationally different in nature.
3. Now for the part where "our brains suck":
Studies show that we are hard-wired for religion. They have to turn off parts of your brain to get to to not believe in a higher power. It’s always sold as “a part of your primitive mind that we should evolve out of”, but in reality, if it was any other part of how your brain functions and it was turned off, we tend to call that brain damage and the results a psychopath. (Side point to show how it sucks to be so absolute in your arguemnts.) Another aspect they have found is that if you don’t fill that void in how the mind works with “religion”, then you fill that void with “superstition”: I really don’t need to define either, the studies do that well enough. The brain is just wired for this stuff and we all have to cope with it in our own way.
All this to say that fantasy and science fiction fills those same voids. (Really, all writing does to some degree.) It’s why we see something in a random book we’ve read and it somehow becomes our mottos that we live by. It’s why so many people fuss about writing characters that glorify negative behaviors. We know, in part, that we are filling a void in how our minds work. Our books become our friends, we roleplay, we talk to characters as if they were real.
But like the pendulum of religion vs. superstition, we have a hard time being in “too many camps”. Camp science likes technical writing and deep human philosophical questions wrapped up in bullet bullet gun gun! It’s harder for them to get into, escapism, the final frontier of fantasy. Just like Regency and Highschool romance doesn’t have the same headspace.
Now, I said all that while being an Egalitarian reader/writer. So, I’m saying all this with a huge grain of salt that has “and?” written on it. I tend towards fantasy that covers questions about the human condition, which means I’m not fitting the divisions. I may not ever get a fan base for being like that. Certainly wouldn’t gain that through writing werewolf romances, either.
)