Forensic Keyword Crawl:
A Note on Crawl Depth: Crawl depth can have a dramatic effect on the time it takes for a crawl to complete, because every level has the potential to contain many more links. The keyword crawler is designed to search breadth first. So the crawler will attempt to crawl all the links at a given level before moving to a deeper level. The parsing and presentation of the keyword matches in the user’s output file can depend on the response time of the web pages (and other factors) but in general keyword crawls specified with a larger (deeper) crawl depth will typically list the keyword matches found on the starting URL pages first followed by the matches at deeper levels.