The pilot light always stays on, its designed to allow your heater to turn on by itself without you needing to light a match every time. if the pilot light is burning you have nothing to worry about. its when the pilot light goes out that a small amount of gas leaks into the room. and by small i mean smaller than a hand held lighter... if you still want to shut the entire thing down you could turn off the gas line feeding the heater or possibly the whole house, otherwise the pilot light will stay on. as far as the actual heater turning off, thats probably controlled by some sort of thermostat somewhere. if you set the temp to very cold it should stay off. if that doesn't work then you may need a new thermostat.