It's a very plausible idea. Dumbledore believes that what is in our hearts, our dicisions, is what defines us. Dementors suck all the good, happy and comforting memories and thoughts out of ones head, leaving only darkness, and if darkness is what defines you, you are lost. Therefore Azakaban intensifies everything that made people go bad in the first place, it doesn't redeem, it corrupts even more. Dumbledore, who values love above almost everything, might very well fear complete blackness of heart most.
But you're right, we probably wouldn't want to know his greatest fear. He's been gaining wisdom for one and a half centuries, hell knows what he must have seen.
Darn coffee, I cannot bring order in my thoughts today!