You should really be using end mill tool holders for milling. The torque is too great for drill chucks. You will get better results machining with surface finish, size as well as longer tool life. Side note..take care with reassembly of drill chuck. Tapers must be dirt and burr free or chuck will have run out as well as taper lock will break and come apart on you more often. If there is damage to the tapers replace the arbor and or chuck with new.