I find it really useful to be able to hear the file I want to import, so I can choose the right file from the Import dialog window.
Maybe I'm misunderstanding you but why not just use Reaper's media browser and preview from there.
+1 Even though I've never knowingly used [Insert] >> Media file. This would add consistency to Reaper: Big pic:
I like the first three options in DarkStar's suggestion, but I would rather like the preview being a tick box below the file list, switching preview on or off when selecting any file.