I figured it out. For others they may want to know...Take a piece of wood or similar, wrap in rag so as not to scratch the towel holder. hold at bottom against the base attached to the wall and tap the wood lightly with hammer. It will slide right off. Mine slid off upward.

Home - Moen