A minor problem is that might, according to some, expresses a lesser degree of probability than may. The big problem is "might have." "I may have left the keys in the car" leaves open the possibility that the keys are in the car. "I might have left the keys in the car" suggests that the catastrophe was averted, as in "I might have left the keys in the car . . . if you hadn't alerted me."
There are cases, as Bryan Garner points out, in which the issue of permission does present ambiguity. He points to "You may not come with me" as a case in which might would solve the problem. In general, however, use might, and especially might have, with caution.