How Does a Search Engine Work?

Working with a search engine is similar to training with a puppy. You find out what they like and use that to get the behavior you want from them. Figure out what they are aware of and what is important to them, then figure out what response you want from them. This is really about training the search engine to fetch.

The first step that a search engine takes is to send out a robot, which is just an automated program, to read each page on your site. As it reads the page, it will evaluate it and try to figure out what it’s about. It won’t keep a copy of the entire page in the search index. It will only keep a score for specific words and phrases and characteristics that it finds and deems important.

This process of reading web pages is called “crawling” a site. After it crawls one page and evaluates it, the robot “indexes” the page. Indexing means that it puts the scores for the page into it’s index, so it can find it when someone searches.

It’s not a human, so it can only guess what the page is about, using calculations based on what it finds. What does it look for and what is important to it? How would you determine what a page is about?

If you looked at a page in your favorite browser and looked at the “source” or the actual HTML that makes up the page, what would you see? Most of us see a bunch of gibberish, but that’s HTML.

The robot sees that same code and tries to figure out what the subject of the page is, based on what’s there. Robots have no nuance or intuition like we do. They are pretty dumb and pretty literal. They only see what they see, which is really just a string of characters.

Robots also look at and track the links on a page and all the other pages on the Internet. They use what they can figure out about these links that are pointed at your page in determining what your page is about also.

Each web page that’s indexed in the search engine has various scores associated with it. These scores are based on, 1. what it finds on the page and, 2. what links it finds elsewhere, that point to the page. When someone types a keyword into the search engine, it goes through it’s index of scores and finds the pages that score the highest for that keyword. It doesn’t read all of the web pages and calculate the scores when someone searches. It only reads the scores that it’s already calculated.

Since how these scores are calculated is a company secret, and probably changes weekly, so we can’t know exactly what they are or how they think. We do know the general idea is that the search engine looks at all of the scores for pages with the specific keyword, then returns the list of pages, the SERPs, to the user in the order of the scores. The page with the highest score for that keyword will be the first result. Our goal is to increase those scores for your pages.