The brake is not fully disengaging and is still contacting the drum. When you play with the pedal you are resetting it.

Check the linkages involved, clean and grease them and also look for bent parts.