Okay keep in mind that this is just atheory that I and my significant other have drawn up. Okay.
Now in the HP series Professor Trewlany seems to be an utter moron who has to DD's knowledge thus far only come up with two genuine predictions in her time as a Hogwarts teacher. Our thought is that when Voldimort attended Hogwarts the first genuine prediction that Trewlany had was that he (Voldie) would see his downfall at the hands of a young boy, who may or may not have been identified as HP. Maybe some characteristic of Harry's that would mark him as the boy or maybe the combination of both James and Lilly. In any case, taken this info, Voldie set out on his mission after school to become Immortal so that he could not be killed by this young boy despite the prediction. So when Voldie finds out about the birth of Harry, which is months later that James and Lilly go into hiding, he comes to kill him.
Just a theory but please pick it apart and let me know what you think.