Search engines

[davedash]

MSN is really good at getting search results for new web sites... or at least it picked up one of my new sites way before yahoo and Google.

I'm kind of curious as to why that is, you'd think all the sites would index pretty much the same things at the same speed. But it might be luck that some of my other sites were indexed more recently.

Comments

[jamminrants]

I did some research on search engine indexing, and there are a lot of factors that play into when and how things get indexed... For one thing, some of the bigger search engines charge good money for quick indexing, so they might intentionally wait to get you to give them money. Also, it could be random - for example, if a site is indexed once a month, and that site happens to link to your site and you got in at the right time in the monthly cycle for a particular engine, then that engine would find your site before the others...

[davedash]

I think it's completely "random" I posted my site to a wiki somewhere, that just happened to get indexed by msn.

It would be cool to have some visual snapshot of what a search robot is doing... you know like some page you could go to and it would like stream a list of pages it's viewing or whatnot... or using some GeoIP stuff to get an idea of what's being indexed.

[jamminrants]

Yes, it would be cool to know what search engines are doing, but part of the nature of search engines is that they need to be very secretive to prevent people from abusing their methods. There are people who make good money trying to figure out the algorithms, and other people who make good money trying to stay one step ahead of those people. Try asking a search engine representative how their system works, and you won't get an answer. Try asking for any of their code, and they'll laugh.