Imo, no it won't. Not if the collet is tightened properly.I agree that proper installation and tightening are very important, but forcing the bit will still vibrate the bit out of the collet sometimes.
I've seen spiral bits on cnc machines take cut's far heavier than you'd ever consider by hand, and the bit's don't come out.