I have had problems with this also. It really would be nice if there was a "repel on this layer" option.