Why is Search Expensive?

“Why is Search Expensive?”

“What do you mean? Google is free.”

Google is, indeed, free to use. But we’re back down to the old adage:

“If you’re not paying for the product, then you are the product”

A quick search ascribes the quote to so many sources that I won’t bother adding a citation.

As illustrated above, search is everywhere. If you are a desk-bound web monkey like me, it’s second nature – easier sometimes than remembering stuff. But search is deeper than that. If you’re looking for the nearest coffee shop or trying to find the size of your car without busting out a tape measure, you’ll be firing up a search algorithm somewhere.

Think about it. Last time you shopped on Amazon – did you drill down through endless menus to reach your desired cluster of products? Of course you didn’t, you used search – like you would on any large store. I don’t think I would even know how to navigate through Amazon (or want to try).

Amazon is free to use. Google is free to use. So why ask the question: “Why is Search Expensive?”

Because search, good search, costs a lot. Let’s look at what search actually costs the people who provide it.

Our ex team member, Cy (retired), wrote the epic tome Royal Navy Officers of the Seven Years War. It’s a bruiser of a book: a 584 page biographical dictionary of commissioned naval officers in the 1700s.

No doubt it features John Manners, but to be sure, you’d either have to trawl through those 584 pages … or check the index.

Indexes are great, almost all databases are connected by numbers. You might be Bob, but to the computer you are user 437 (or example). The computer will store all your orders and your preferences against the number 437. Your user record will likely hold a number called something like ‘photo_id’. The server won’t look for ‘bobs_profile_img.jpg’, it will look for the photo with the indexed ID matching your photo_id value.

Numbers are much quicker to look up than text. Numbers organised into an index, even quicker.

For both humans and computers, checking through text for individual words takes a long time. Or, in the case of computers, a lot of processing power. In the world of fast, cheap or good – to get fast, you need to forget about cheap.

Indexing the massive volumes of data that must be searchable at all times takes a huge amount of processor power. Indexing involves parsing data, extracting relevant information, and organising it in a way that can be quickly searched and retrieved. This is not a one-off process; as new data is added, removed or modified, the index must be continuously updated to ensure the accuracy and relevance of search results.

The complexity of this process increases with the size and variability of the data. Data can come in numerous formats (text, images, video, audio), and processing it to extract meaningful information for search queries can be computationally expensive. For instance, search engines often deploy natural language processing (NLP) algorithms to understand the context of queries, which requires significant processing power and, by extension, cost.

Years and years ago, I was working in Our Price Records. A client came in and said: ”I heard something really nice on the radio, the song went ‘ticky’ something …”.

Twenty minutes later, she left the shop with Tanita Tikoram’s Ancient Heart album in her bag.

It was the 1980s – I was young, helpful and still possessed of an agile mind … we worked it out. But it took some effort.

People cannot type, people cannot remember, people cannot spell – it’s a wonder we can communicate at all. To find good results when our stubby, cold fingers type “em,rgeny heath car” on a windy night, the search algorithms needs to compensate for our shortcomings.

It may not seem it, but the search market is competitive. Google has held the market lead for a long time. The verb ‘to google’ has become synonymous with ‘to search’. But most of us own what we call a hoover when, in fact, it’s a Dyson, a Henry or somesuch.

Loyalty on the internet is wafer thin – a few duff or slow search results and users will flee.

So, as user expectations change and search technology evolves (think of advancements in artificial intelligence, machine learning and Natural Language Processing), continuous research and development (R&D) is necessary to stay competitive and improve the quality of search.

R&D is costly.

Engineers, data scientists, and researchers are needed to build and refine the algorithms that can interpret user intent, rank results, and filter out spam or low-quality content. Developing the algorithms that power search engines and platforms is another major expense. A simple search function might seem straightforward, but providing accurate, relevant, and personalised results is a highly complex task that requires constant improvement and adaptation.

To provide instant, relevant results, search engines and platforms need extensive infrastructure. This includes vast data centres filled with powerful servers that must be maintained, upgraded, and cooled constantly. The hardware required is not only expensive to procure but also to operate. As the amount of data and the number of users grow, these infrastructure demands scale, requiring more servers and more efficient technology to handle the increasing load.

While a lot of queries are common (e.g. ”best curry nether edge”) and indexes can be cached to some degree, the target is always moving (Curry Tonight went bust). Those cached search results must live somewhere and updates must run frequently.

And then there’s always some beggar who goes looking for long dead navy officers. Hardly worth keeping a cached index of that – just fire up the big guns and look. Don’t forget the whole process needs to take moments.

So, addition to purchasing and maintaining servers, there are costs associated with data storage. The vast amounts of data that need to be indexed, processed, and queried require significant storage space. High-performance storage solutions that can deliver fast access speeds and reliability are essential, further driving up costs.

Still looking for that coffee shop?

Modern search is expected to be not just fast but also highly relevant and personalised. These days, someone entering a query like “train times” will want results for the local area and their preferred transport methods, tailored to the time of day. To achieve this level of personalisation, search engines and platforms collect, analyse, and store vast amounts of user data, including browsing history, location, and preferences.

Analysing this data to deliver personalised results is computationally intensive and expensive.

Search engines and platforms are often used globally, meaning they must be able to scale and provide consistent performance worldwide. This requires robust, globally distributed networks to minimise latency and ensure fast response times, adding significant costs in network bandwidth and content delivery networks (CDNs).

Supporting multiple languages, dialects, and regional preferences further complicates the process. Each language requires its own models and algorithms to ensure that search results are contextually and linguistically accurate. The effort to localise and scale search for different regions around the world can become costly.

Search engines operate large data centres that consume significant amounts of electricity. The power required to run servers, cool data centres, and ensure redundancy (to prevent downtime) is immense. This energy consumption not only translates to high operational costs but also has an environmental impact, pushing companies to invest in greener, more efficient technologies to mitigate their carbon footprint.

Companies like Google and Microsoft have been investing in renewable energy and improving their data centre efficiency, but these initiatives come with additional upfront costs. The drive for sustainable operations can make search more environmentally friendly but, at the same time, adds to the overall expense of maintaining search capabilities.

While much of the cost of search lies beneath the surface, the user interface is a critical aspect that also demands attention and resources. The search experience must be user-friendly, quick, and intuitive. Designing and maintaining an interface that meets these standards requires ongoing UX/UI research and design, testing, and optimisation.

For mobile applications and websites, the interface must be optimised for performance, speed, and accessibility. Ensuring that users can search quickly and easily on any device requires cross-platform development and quality assurance, which adds to the overall cost.

In the vast majority of cases, advertisers.

Google is actually an advertising billboard with a search engine attached … and you, dear people, are the fodder.

Consider the current (Summer 2024) iteration of the world’s most popular search engine. The quantity of paid advertising on the screen dwarfs the volume of genuine search results. Like US public access television, it’s hard to tell where the advertising ends and the content begins.

We’d argue that this is self defeating – as we type, people bemoan the fact that Facebook is nothing but adverts. Ultimately both platforms will suffer if the reason why people use them is lost in the advertising.

On a large e-commerce store, search will be factored into the price of the product.

In most cases, it is. Computers are amazing and the search provided by self-hosted sites is largely adequate. The traffic is low, the results fair and your site does not stand and fall on the search. You’re hosting fee probably covers it.

However, be warned, WordPress search is only OK. WooCommerce search is rubbish. We have spent significant time tuning databases so that even a decent-sized store got decent results. You can buy in better, dedicated search for WooCommerce but it costs.

If search really takes off on your website, your hosts will want to put your hosting price up. They may not call it ‘search’ but that is what it is.

Wix, Shopify, Webflow … all those subscription services, you can be sure a good wedge of the subscription price is spent ensuring the servers have enough ‘grunt’ to keep the search flying.

If your customers cannot find your products on Shopify, you can buy in an external search service. Prices head into hundreds of a month. It’s a lot, but remember Amazon? Search drives sales.

If you don’t mind the advertising and you don’t own a website, not much really.

But search drives sales. If your business is growing and your site getting busy, your search may be letting you down. At Little Fire, we’ve improved existing website searches, implemented ultra-fast third party solutions … we’ve even written our own.

We can help. Contact us.