Reading through all of your comments I realized my idiocy. I was never seating the collet in the nut, but rather just inserting the collet in to the spindle, inserting the bit, and then tightening the nut. So when reversing the process the collet nut would come off the threads but the bit was still in the collet, which was still in the spindle.
Yes, I’m one of those idiots that doesn’t typically read the manual. Maybe that should change.
Thanks for all the help.