What is Literary Fiction?

Many editors believe that one of the primary differences between literary fiction and mainstream fiction is that mainstream fiction tends to have a stronger emphasis on story or plot than on character.

This does not mean that plot is not important to literary fiction; narrative movement and a good story line are crucial and must include the elements that we commonly associate with stories.  The work must be kept together from start to finish.  

Some elements of literary fiction:

Introspection
Complex and layered, with in-depth character studies
A focus on characters that drive the plot 
Usually enlist emotional involvement in the reader
Are primarily based in language and metaphorical
Concentrate on the senses

Some examples of popular literary fiction:

The Stranger by Albert Camus
A Tale of Two Cities by Charles Dickens
Mrs. Dalloway by Virginia Woolf
Crime and Punishment by Fyodor Dostoyevsky
East of Eden by John Steinbeck
Catch-22 by Joseph Heller
Cutting for Stone by Abraham Verghese
Corelli's Mandolin by Louis De Bernieres

Whether or not literary fiction is called a "genre" is debatable.  It's all fiction, excerpt the literary style tends to be lyrical, takes risks and concentrates not on the climax but all the drama, before and after.

If you enjoy intrigue, plots and subplots, and novels filled with action, then mainstream fiction is what you want.