There has been discussion (notably at
tor.com) regarding the relationship between mainstream novels (referred to misleadingly as "litfic", but I'll come to that) and Science Fiction. Aside from revealing the deep insecurity that many commenters have regarding the relative status of the two fields, I've noticed several things that seem to be, well, just wrong.
First, there's a sense that novels that are more serious and less funny/optimistic are more likely to be literary (the deck was stacked in that the article triggering the discussion was on dystopias). Look, the "Great Tradition" of the novel begins with comedy -- Pamela, Tom Jones -- and continues through Jane Austen. Even Ulysses is a fundamentally comic work.
Secondly, using the dystopia as the model for "genre in the mainstream" is not in fact typical. My guess would be that there's far more crossover into the mainstream in works which could be broadly described as "Magic Realism". In fact, I'd class the dystopias of Huxley and Orwell (and Zamyatin) with other works of the mid-century making sharp critiques of the emerging industrial society (such as Modern Times), and not with SF/F at all, which is a different reaction to the same set of stimuli. Some SF picks up on the dystopian themes (cf. Camp Concentration) but that's as much SF crossing lines into the dystopian genre as the reverse.
Thirdly, what gets tagged as literary fiction is not a genre as such: it's fiction which tends to have a higher degree of attention paid to language and construction. And there, there is a very grey area, since there are plenty of books which hover near the area but are not (yet) clearly in or out: it really takes about 50 years to start to separate out flashes in the pan from the real gold. Thus, the Dance to the Music of Time is almost certainly in, as are The Lord of the Rings (regardless of the disdain of a persistent type of "literary" reader) and The Horse's Mouth, but it may be a little early to tell for Possession and some of David Lodge's or Robertson Davies' work. Wodehouse has long since made it in as a craftsman (some of his novels, like Laughing Gas, arguably cross boundaries into SF/F). (Note that the above are all comic writers/works.)
Within clear SF/F, I think that there are strong candidates for whatever becomes the equivalent of a late 20th century / early 21st century set of literary "standards": Little, Big is obvious, as is Jonathan Strange and Mr. Norrell, and Stephenson seems to be knocking on the door for what Frye called "Menippean satires" (although he meant something rather different than what Menippus produced). Newer authors like Valente and Mieville make a strong showing on stylistic grounds