Once upon a time, search spiders would crawl the web largely recursively

[zihadhasan019]

Hitting a homepage that had been submitted to its index (remember way back when search engines had submission?!), then crawling in an outward fashion based on the links they discovered there. That hasn't been the case for a long time, and as we all see with crawl paths (if you're looking at the requests Google/Yahoo!/Bing make to your domain), multiple entry points are nearly universal and crawling pushes "outward" from those priority URLs.


It looks a bit like Minesweeper, right?Spider C israel email list rawl Priority Paths Graphic Finally, I've got a graphic to help understand how to positively approach these problems and solve them. Methods to Improve Crawling, Indexing & Ranking There are certainly more recommendations that can be provided around these issues, and I look forward to a discussion of them in the comments. p.s. I covered site architecture and navigation in a good bit of detail at the PRO Training this summer, but I like this image format so much, I think I might re-craft something new for next year.





It feels like structuring sites properly is still a big pain point for SEOs (but possibly that's less to do with lack of knowledge and more to do with lack of influence during the design phase?) As we rapidly approach the end of 2009 and opening of 2010, we've got a much anticipated index update ready to roll out gang. Say it with me "twenty-ten". Oh yeah, I'm so gonna get a flying car and a cyberpunk android .