On some models the collet stays put on purpose. You don't have to remove the nut either. But if you really do have to remove the collet and it is the type that is supposed to come out (maybe you need to use the 1/4 inch collet) then just stick a bit into it without the nut and give the bit a sideways whack with a piece of wood. If the collet has been pressed in tight by the nut without a bit in place, then you won't be able to fit a bit into it, and maybe the collet is ruined. Find the largest diameter shaft you can to fit into the hole and pop that sideways. Good luck.