Do you have the right size ground wire, relative to your power wire? If your ground is smaller when you turn it up there is alot of power it needs an exit and if the wire is too small it travels elsewhere maybe your rca, causing your amp to protect.

If your amp is a 2 channel you might be using the left positive and right negative or vise-versa. check and make sure your only using right or left and not both, as it will cause your amp to go into protect