Best thing to do is too take into a bike shop and have them take a look. It really isn't possible to tell you what is wrong and how to fix it without seeing it.

Taking the brakes off your bike should have no effect on steering as the two things are not connected, so your assessment that you messed things up sounds pretty accurate.