I'm not sure that's true.
Search engine robots can't scan directories they can only find things referenced in web page sources. If there are no links to a page it will not be found. That's my understanding.
This may also be interesting: https://moz.com/blog/12-ways-to-keep...search-engines
As one site I visited pointed out, don't put references to stuff in robots.txt that you don't want people to know about - because anyone can look at the robots.txt file, not just indexing spiders.
Paul
Bookmarks