The adapter is probably stuck by a protrusion from the adapter. Try gently flexing it up and down as you push/pull it out.