I am not too sure how you can use the Eraser to retrieve colours from a B&W image.
I think what you might have done is to have the original image on it's own layer, then duplicated that layer in the Layers Palette and converted that to black & white by either Desaturating in H/S/L Hue Saturation Lightness, or used an Adjustment Layer and the Channel Mixer.

I prefer to use the Desaturation method, adjusting the Saturation to 0.
Then I erase the parts of the B&W image that I want to see in colour, showing the original layer.
Use a Eraser Brush on, say 75% Opacity and set the Hardness to 0.

This is the easiest way to 'retrieve' the colours.

Grenou